The you're downloading the wrong types of programs.
I haven't run a task killer in over a month and the ONLY time I see a significant slowdown is if I just finished updating a lot of apps, or I'm trying to switch between apps and flash content on my browser. My phone had MORE slowdowns when i used advanced task killer. I found the less I used it, the less I had, so now I never have it running, and I'll most likely uninstall it.
If you're concerned about it, download "Spare Parts" and set the activity process monitor (the built in memory mananager) to aggressive.