I am in the camp of the people that this is driving them crazy. I completely understand how Android is managing these apps. However here's a couple important points.
Those apps are taking up RAM. You can tell by the fact when you open them from the task list it's instant and you start where you left off as opposed to taking time to load.
Yes, it doesn't matter they are taking up RAM because Android will truly 'close' the app if it needs to, to free up RAM for another app.
BUT, If you have a phone with lots of internal memory like mine (the G4) that means that the background app list or task list or whatever you want to call it can grow very large over time. In a matter of a few days mine will have 25 or more apps listed! As other users have pointed out, when you have that many, flipping through to find a certain app is actually more cumbersome than just starting it from the regular menu.
So that leaves the question, what's the point at all of having the background app list. I'm not talking about the launch time benefit, that benefit you get no matter how you launch the app if it's already in the background I'm talking about the practical use of the card interface when your phone is saving a gazillion of your last used apps - WHICH IN MY CASE (and many others) MANY OF THOSE APPS I USE MAYBE ONCE A MONTH!
The benefit is good, but the interface excution of this feature is obviously not well thought out. Surprising for such a big company like Google with som much brain power and development resources.
In Settings > Apps there should be a setting, 'Close inactive apps after X days' So if you specified 5 days, the O/S would remove the app from the list if you didn't open it for 5 days. That way your card list of apps would truly be your apps you use frequently.
I also would note that I too have noticed my phone gets slower when I have 20+ apps listed. But as the other user pointed out, the reboot is what might be the key to speeding things up, not just closing all the apps.
I like the fact though my G4 has a 'Clear All' button. I guess LG agrees more with me than they do with Google on this subject.