I dont have problem with lag on the phone, but i also think lag has got to the point of being subjective (like recoil, feels different to different people).
There are apps that take a moment longer to load than other apps, but its not the phone its the app, because it happens on different phones ive used, and to blame the phone would be nonsense.
To me the note4 is very fast, but then again i dont expect it to read my mind and open an app before i select it lol.
The only time i see a short delay is when using the priority contacts bar in the text message app, i can actually feel an ever so slight "lag" from the cursor going from people list to message portion..
The phone is super fast, your results may vary.. esp when using some slower apps.
As far as what u lose, i have read u lose some spen stuff, things that are specific to the note line, because the spen is part of touchwiz.. some mind and some dont depending on how much they use the spen. I use the stock launcher because touchwiz isnt as bad as it used to be, and i use the spen quite a bit.
Good luck to ya!
Verizon Note4 w/ Otterbox Defender