I just switched yesterday. Used my jump upgrade. So far don't regret it. On paper the N6 is larger and even looked a bit larger in pics online, but in reality, it's nearly the same size as my note 4 was and honestly felt better in hand. I've had the note 2, 3 and 4 and all were nice devices but I really didn't use any of the fancy features. The spen was the only thing I used and even then, rarely. Most common use was cropping a screenshot. Didn't use multi window or the floating apps and was never a huge fan of touch wiz. The note 4 particularly annoyed me with its excessively large notification shade touchwiz footprint; with the quick settings and the stupid s search and s share buttons, half the screen was useless for actual notifications. Plus the device was a tad laggy at times.
Havent had a nexus phone since my nexus one, and I'm loving this thing. Lollipop runs and looks great. No bloat, no lag. People say the screen is worse, but it looks damn good to me. Maybe if I had the same pic on each device in front of me, I could spot differences, but honestly, this screen is great. I'm not doing photoshop and print work on this thing.
Couldn't get 5.0 to update on its own yesterday so I side loaded 5.01 and 5.1.M then did a factory reset. Running like a champ.