If money is not an issue then go for it. Save the GS4 as a backup.
Google Nexus 6P
I went from my trusty GS4 32GB to a Note 4 in late 2014 and have not looked back! I was also concerned WRT pocketability of the Note, but here I am nearly 2years with it and not a problem, I am even typing on it one-handed on this post.I will keep it as a backup. The gs7 is the idea size, but I would like to try the Note with the s-pen. But, it doesn't look pocket friendly.