I think for those with a Note 4, it may not be worth the upgrade, but I don't think its a downgrade from the Note 4 either
Ram
I have a device with 3GB of ram (S6) and a device with 4GB of ram (Asus Zenfone 2), and the difference in performance is quite noticeable. Yes its true they have different OS but you can tell the 4GB works much better especially when opening a bunch of apps. I use my Zenfone 2 mainly for games, and was able to run Tomb Raider and Asphalt with no lag at all. In my experience, 4GB is noticeably better than 3GB
Processor
As for the Qualcomm v. Exynos, the benchmarks show that the Exynos performs much better than Qualcomm. Qualcomm even has its own issues with overheating, which they may or may not have fixed. I think the Note 5 will have much better performance than the Note 4
Ultimately, I see it as an upgrade from the Note 4, even if not by much
- Better Camera (Same as S6, which performed better than the Note 4)
- Bigger Ram
- Likely longer battery life with the Exynos processor
- Likely enhanced S-Pen
- Exynos benchmarks much better than Qualcomm