Not a Note 5 user here, but going from the Note 4 to the regular S7 works for me. I was really tired of the Note in my pocket. I'll probably miss the S-pen sometime, but really didn't use it that much.
The biggest difference to me is the size. I much prefer the S7, but I do wish it had an S-pen. The S7 is the nicest phone I've ever touched. I can't get over how awesome this device is in the hand, but that's just my opinion. I'll probably never go to a Note sized device again. At least not where they're at currently.