The recent tasks list can be confusing. Here is some info I learned on it a while back...not sure if this is entirely still accurate with the latest versions on Android, but based on my own experience, this is correct:
1. The recent apps list is just that, "recently used" apps. Just because an app appears in the list, it DOES NOT mean that it's actively running or doing any processes in the background, it simply means that the app has been used recently. Any app that appears in the list *may* be completely shut down and not running at all...or maybe it is. In other words, it is not a list of actively running apps.
2. Having said that, however, swiping apps away from within the recently used apps list will close them *IF* they are still running, and is essentially the same process as hitting the "back arrow" key to exit out of a running app (i.e. a "graceful" exit, as opposed to a "force stop"). If an app is designed to sync regularly or start itself to run background processes, then it will continue to do that even if you've removed it from the recently used apps list and forced it to exit...so even if you swipe an app away, it is possible that it will still be "running" in the background if you check it's status in app manager.
Hope that makes sense.
To the OP, it really isn't something you should worry about too much or allow yourself to become obsessed with. Android does a pretty job of task management on it's own. I personally don't use the "recent apps" feature very often, and rarely even open it...nor do I ever go into the application settings to shut things down manually. I do try to reboot my phone every few days just to keep it "fresh", but honestly I'm not even sure that's 100% necessary unless things start to run sluggish. My phone has fantastic battery life too.