I've had my note 5 since launch as well and echo your experience. I've never owned a stock Android device and after reading a few glowing reviews of the 6P, I figured I'd give it a test drive. Fortunately I was able to simply move the Sim card from my note 5 into the 6P seamlessly.
I've used the 6P for a solid week now. I'm impressed with the styling, the look, the way it feels in my hand. The screen is simply beautiful. As I mentioned earlier, the whites are sharp. There's absolutely no lag at all. This thing sings as it goes to work!
At first I thought I might end up selling the Note 5. But I'm going to hang onto to it. I have 30 days to test drive this 6P. As I've learned from the stock community, early updates and lack of carrier "bloat" are two big selling points for going stock. If those items are important to you, the Nexus line is a no brainer. Those have never been important to me. And that's fine either way. Stock is more barebones. I like my phones like I like my cars, packed with necessary and uneccesssry features.
A lot of people make a big deal about updates, bloat/"uneccesssry" features and being unlocked. Not tied to a carrier. Those are all absolutely legitimate items to those that find them important. But if they are not critical to your buying decision, then you'll probably have a difficult time justifying the switch.
Regarding the Spen. Another uneccessary item. But I enjoy using it daily whether for production or to kill time. I've found great uses for it. The ability to pull it out and write a quick note on the black screen of my phone is incredibly useful.
I love this Nexus 6P. I really do. There's more of an emotional connection with the 6P whereas the draw to my Note is function and power. Sort of like you'd pick the Nexus 6P to take out on a date or to a house party and the Note 5 you'd call over to watch football or fix the roof. The 6P easily trumps my note 5 on design, feel and looks. Note 5 has a gorgeous glass rear panel. But it's an uncomfortable finger print magnet. I had to put a skin on it. This 6P feels perfect in my hand naked.
A huge plus for the 6P is Marshmallow out of the box. And is sweet! Not sure when Samsung will get it. They have 1000 Galaxy models to pore over first I guess. But Marshmallow is worth the price of admission because I can call people or ask my 6P to play music or compose a text or email without ever touching my locked phone! I use that feature so often I'm afraid I might break it! And it's flawless! From across the room I just say, "Ok Google" then say a command. My Note 5 will eventually receive Marshmallow. When I don't know. But it has me spoiled.
The front facing stereo speakers on my 6P
are
the
best
ever!
Boom sound on the HTC might have slightly richer sound, slightly, but nothing is louder.
You might need to slide over on that fence and make room for me! Lol
Google and Samsung have hit it out of the park with the Note 5 and 6P.
I definitely encourage you to take a look at the 6P. I would love to hear your impressions.
The above are just from my experience. Everyone's story is different. I acknowledge and respect that.