As you can see from my phone history, I have owned phones with non-removable batteries. Within 6 months on each phone, I found the battery life insufficient for my needs (which, admittedly, are probably heavier than many other users' needs).
I'm not going back to that. I'm just not. A year of battery anxiety in meetings was enough for me to look at anything with a sealed battery (including the Nexus 6, which I actually pre-ordered then canceled because I realized it was just another iteration of the Moto X) and have a big ol' helping of "NOPE!" with a side of nopelsauce.
The Note 4 is my first Samsung phone. Moving the back button to the right side drove be crazy for about a month. I am not a big fan of touchwiz, but I learned to like it. It's actually not that bad once you get used to it. I hate the delay in getting OS updates. I fell in love with s-pen, but it's one of many factors in my next phone.
Point is, the Note 4 is a compromise. It was the best available large-screen phone with a battery I could swap out available at the time. It has several features like S-pen that I like and make the Note 5 more desirable to me. But not "more desirable" enough to give up a battery I can swap.
I gave up Moto X's take on voice activation, their "twist and shoot" camera feature, smart unlock (which works with on the Moto X series with MobileIron but not on the Note 4), fast OS updates, their quirky startup animations and 3D immersive films, and a bunch of other things to get a swappable battery back. If it comes down to it, I'll give up S-pen for the same reason.