It is the way phone operating systems are designed. Basically, it preloads the apps you use the most into memory, so that when you go to use them they are quickly available, while not slowing load times if you load an app that isn't preloaded. It also helps save on battery usage, memory being "full" doesn't cause more battery to be used, and it saves battery by not loading and unloading your commonly used apps into/out of memory each time you use them.
Which version of N are you using? Preview 2 had serious memory leak problems that would cause the phone to slow to a crawl. Preview 3 (the latest) has fixed this, I haven't experienced any performance degradation problems on 3.