I will consider the S6. To be very honest, the Lollipop update to the N4 as far as performance goes will really have to move me. Performance is very big on my list and I just don't care for the sluggishness of the Note 4. It's not terrible, but compared to my Nexus 5, there's no comparison in how nimble the Nexus 5 is. I know some will be quick to say that because of the Note's feature set, that the performance should be overlooked, but for me it can't. I'll still consider another Samsung device though (S6) because first and foremost, I love the camera!! A f/1.9 aperture on a phone is just crazy and I do quite a bit of cell phone photography. Secondly, the Note 4 is a behemoth. I've had mine about 4 months and I'm still not happy with the size and weight. When it's in my pocket, I always know it's there. I wouldn't mind downsizing.
But nothing is definite. I'll see how Lollipop helps out the performance of the Note and then I'll do some research on the S6 to see if it's the right choice.