I switched from Note 4 to the 6P.
The 6P really smooth. The N4 was pretty smooth, but some things, like the Recents display, always lagged for me. Not on the P6. And just makes it a pleasure to use.
The one thing I miss from the N4 is the front button to wake the phone. The back facing fingerprint reader is great on the P6, but not when the phones lying on your desk (often for me).
The numbers row on the keyboard is easy, just switch to SwiftKey keyboard.
No big screen aids, like the swipe-in-out shrinking that Samsung has. I only used that on occasion, but was good idea Google should incorporate.
No milti-window, but the few time I used it on the N4 was to make task switching easier by getting around the lagging Recents list. Supper reliable and fast on P6, so less need for the multi window (for me).
Front speakers are very nice. No more flowing the phone on its face. No more cupping the back speaker.
Physically the N4 was a buy easier to hold. The P6's thin, all metal body is a bit low of friction. Also the P6 volume buttons are a bit too low and too easy to hit (I've had not issue with the Power button that some reviews had noted).
I think it's a good upgrade from the N4. Especially given the reasonable price. Plus, I'm switching to Project Fi, so nice savings over Verizon.
Good luck.