I too was very happy with my Note 2, but found the Note 4 has some very compelling advantages. The first is the screen resolution and quality; it's vastly superior IMHO. It's much crisper and brighter and easier to read (ex. small text on webpages). You really have to see it first hand to fully appreciate it. Next is the improved S-Pen which now has a soft tip and comes with replacement tips if needed (compared to the non-replaceable hard tip on the Note 2) and can now be used on the capacitive back and recent apps buttons (no more having to use your finger). There are also numerous software improvements for the S-Pen as well that are far more useful than on the Note 2. Finally there's the vastly improved build quality with the metal bezels/sides and soft touch back (sans the cheesy faux stitching of the Note 3).
It's been almost three months now and I have zero regrets (even in paying full retail to keep Verizon grandfathered unlimited data).

I still have my old Note 2 as a backup and it feels very outdated now when I attempt to use it.