It's a shame because pre-lollipop the phone was damn near perfect!!
No it wasn't. The GPS was hit or miss, opening and working with resource intensive apps like Gallery and S Note were painfully slow, battery life the way I used it didn't get me through the day, and the chamfered metal edges looked beat up after three months of careful use. I had all the Notes except the original and they were all strong phones but each had their strengths and weaknesses.
The Note 5's weaknesses the way I see it are the loss of some settings and features (EG: Side Panel), the flexibility of expandable storage, the peace of mind of replaceable batteries, and the fragility of the glass back panel (its already cost me $75 to replace). Its strengths? It's wicked fast no matter what you throw at it, I get 30% better battery life, fast wireless charging is brilliant, Samsung Pay is a killer app, Samsung’s apps and the UI are for once streamlined and consistent and its design is beautiful.
When I traveled internationally with my Note 4 I relied on a small 2A 10K mAh Samsung battery pack rather than buying and having to charge replaceable batteries that you realize you forgot to charge when you insert one when you need it most and it's empty. I'll rely on my battery pack with the Note 5 as well. As for storage I loaded all my entertainment for 14 hour plane flights on my 64GB Wi-Drive which lets me access it or stream it to both my phone and Note 12. Same thing for large work files. Nothing's changed there either.
Maybe some here need SD cards and replaceable batteries but it's not like there aren't plenty of workable alternatives available. Apple and Nexus users were making it work for years when there were fewer alternatives that cost more.
People saying previous Notes were better than the Note 5 haven't used one for an extended period of time. And if you absolutely need replaceable batteries or expandable storage there's no need to consider the Note 5 in the first place.
Almost everything being discussed here is a YMMV.