I had an S3, then an S4, and now the Note 4. No regrets yet.
I've only had it one day, and that was working and sleeping, so really only played with it for 6 hours or so, but this is what I have found:
The good:
1) It isn't that much larger, I can still hand hold it.
2) The display is AWESOME.
3) The S Pen works well with OneNote, my preferred Note taking software. And of course all of the other handy things the pen does, if I remember to pull it and use it.
4) It feels good in the hand. I hated to put a case on it, but I am.
5) The display is bright in sunlight. Which is good since I use it as a GPS.
6) The speaker is louder then my S4.
7) The camera takes better low light pictures then the S4.
The bad:
1) I also bought a spare battery. Good thing as I have had to swap them a number of times in the day that I have had it. Hopefully it is just because of the constant on time, and constant app and document loading that I have been doing.
2) Related to #1. I clipped it to the handlebar on the motorcycle, plugged it into the VERY WEAK charger that charged my S4, and rode for about 15 minutes running the screen on and the GPS app. On the weak charge, in 15 minutes, the power went from 81% down to 78%. Not a good sign.
3) I got white, since I have always had black or dark colors. Not sure I like the white.
Needless to say, I am concerned about battery life. I have ordered a few 12 volt, 3 amp, chargers. That may/should correct the GPS drain issue. but I need a few more days to find out.