First, a quick hardware comparison:
Samsung Galaxy Note 4 vs Samsung Galaxy Note 3 - Phone specs comparison
Big beats: The Note 4 has a higher-resolution screen, a higher-resolution pair of cameras, optical image stabilization on the camera, a better/faster processor set, better noise cancellation, a fingerprint scanner.
In addition to those, there are improvements in S-Pen and its features, and of course it will get Android upgrades for a year longer since it's a year newer.
The Note 3 is still an excellent phone. If none of the above improvements excite you, I'd wait until the Note 5 comes out and choose between that or a (by then) greatly-reduced-price Note 4.