I think this is an Android problem (always was, it's just less bad now), because I have a Moto X and the first week, before I'd downloaded any unnecessary apps, it was zippy fast on almost everything, but then when I finally downloaded all the other apps I use (games, news, books, etc), things started scrolling jittery and jerky. I then noticed only some apps did this (like Twitter) and some apps didn't (like the Play store), so either these apps are not optimized or the others are optimized better. It seems it only happens when it has to load pictures on a list, as it pauses just before it gets to the picture then continues.
On iOS (iPhone, iPad, iPod touch), usually pictures on a list lazy-load, meaning the list still scrolls smoothly but the pictures load as you scroll rather than making you wait for them to load (or they wait until you stop scrolling). I think unless an Android app is optimized to do this (or it loads everything into RAM at once) it will be jittery. I do think Android can be programmed to get around this (meaning preventing app developers from having to do this) or find some middle ground to not make the phone seem laggy.
I did read that someone found the Amazon apps ping the CPU too much and too often, so I uninstalled that and things have definitely improved (uninstalling the games I played made almost no difference, and I had a half a dozen graphic/memory/CPU-intensive games), though I still notice the jitteryness and for some reason the pull-down bar is affected dramatically as well.
I just think Android is not optimized for scrolling lists with images (and I have no idea why more apps on the phone would slow down the pull-down bar but not the homescreens regardless of how many notifications you have up there), and also developers' apps are not optimized for this. However, I do think both can fix this so that it's not an issue even on a slower phone.