Here is the thing... I bought the note 4 on day one. Loved the phone. Then recently I bought a BlackBerry Z30, because I missed my blackberry storm from years ago. I couldn't believe the lag on my note 4. It just made me realize that TouchWiz needs to be revamped.
Posted via the Android Central App
The only time I saw anything "lag" was when my phone first updated to Lollipop 5.0. When swiping between the Home screen to another screen, I would see a delay and then sometimes the screen would jump to the next screen. It wasn't smooth at all. I also noticed a delay going into the app drawer and say, selecting Settings, the Settings window would delay before appearing. Not a long delay, but it definitely was there. I had other problems with LP as well, like dropping calls, my phone app closing down for no reason, frozen screens, eating my battery, All problems disappeared once I cleared the system cache (in recovery.... wipe cache) and doing a full reset. However, I really didn't like the new LP interface, so I reverted back to KitKat and that's what I'm running now. I'm much happier on KK! I've tried other launchers too, but TouchWiz seems to work fine for my needs.
By all means, if you feel you need a change, then that's what you should do. Doesn't hurt to try other things. Good luck!