Interesting dilemma. I went through this myself a couple of weeks ago. My push toward the Note 4 was the sim card. I wanted a phone that used the Micro Sim Card like my Galaxy S5 as I wanted to be able to switch back and forth between the S5 and the Note. The Note 5 uses a Nano Sim Card.
Then I found out (here at Android Central) about sim adapters and I found the best sim adapter was already in my kit from Tmobile. It is the Trio Sim Card from Tmobile and allows me to easily switch my sim from the Note5 to the S5. I am ecstatic that I didn't have to downgrade to the Note 4. The truth is about the interchangeable battery is that while
thought I wouldn't be able to live without an interchangeable battery that was just nonsense when I shined the hard light of reality on the subject.
I had my Galaxy S5 for about a year with one extra battery. Not once in that entire year did I ever need to change the battery. So that knocked the "interchangeable battery deal breaker" out of the picture. Not an issue especially since I picked up a 15,200 mAh power bank by Choe from Amazon. This will do a FAST recharge
if needed of my Note5, 5 times on one charge. However, the Note 5 is less power hungry than the Note 4 in the first place. Now the battery issue was truly eliminated.
Then regarding the absence of the micro SD card, I picked up a Dash Micro SD Card Adapter from Amazon for $13. It will adapt up to 128GB micro SD cards to my Note 5 with a tiny accessory that simply plugs into the USB slot. Bingo! There went the so called "micro SD card deal breaker" issue.
Then the only thing remaining that could even remotely be considered a "con" in upgrading from a Note 4 to a Note 5 is the absence of the IR Blaster. Again, for me, this was a non issue. All of my equipment at home is controlled by Logitech WiFi remote control which offers an Android app that works perfectly without IR.
So bottom line is I went with Note 5 and I am very glad that I did. The feel, the horsepower, the improved software - everything is just better on the Note 5. I have no reason to ever look back.
The only valid reason I can see someone might need to go with a Note 4 would be cost. If the prospective buyer cannot afford the Note 5 then by all means, the Note 4 is a good alternative. But all these other BS complaints about the battery, the SD card, etc., are just that - BS.
I believe that the technological advancements on the Note 5 are worth every penny of extra cost over the Note 4. In fact, in my business, the ability to take notes down without having to engage the S Notes app has paid for the cost difference already. I just pop the pen out and start writing, try doing that with any other smart phone on the market, it won't happen.