I'm experiencing similar issues with my T-Mobile Note 4. It's very random, because sometimes it performs snappily, and other times it is actually slower than my Galaxy S4. I see a lot of lag when moving between apps occasionally, however my worst issue is dealing with the keyboard. I had initially been using google keyboard and would see a noticeable lag that affected accuracy, so I switched to the stock keyboard. The stock keyboard was great for a few days, but is now even worse. It will freeze as I am typing a message, and the words I was typing, it will actually put into the message 2-3 times (forcing me to go back and edit it). It will also experience the same accuracy/lag issues that I had with Google Keyboard.
Another thing I thought was weird was when I tested out ART vs. Dalvik for a few days. I switched to ART and rebooted and waited for it to compile all the apps. However, after this, every time I would reboot the device I would have to wait for it to recompile about 140 apps, every single reboot. Of course, each time this happens it runs down the battery a lot, and the phone would get very hot. So, I eventually switched back to Dalvik. Someone told me that this isn't supposed to happen, and ART is supposed to compile all the apps once, but not on every reboot. So, I'm not sure what's happening. I want to love the device, but now it's too late to return it and so I'm thinking I'll just try to swap it with the retailer.