Here is my 2 cents. My buddy has the One I have the S4.
I've noticed lag on my phone with the touchwiz. but thats probably more due to the fact they rushed to get 4.2.2 on it more than anything. Everything else works beautifully.
Problems, I've noticed already with my buddies One, which I only see every so often.
Microphone issue. Sometimes the mic picks up lots of static. So say you are doing a google search... the mic will pick up static and not stop so your voice search doesn't do anything. Kind of annoying when you need an answer right NOW
Camera,great in low light, but after that its basically a joke. Weird thing is, when you are going from a dark area to a lite up area the camera does some weird red color on all the black.. really funky.
but, that can all be fixed in updates, I would hope.
The biggest problem is signal... the s4 just gets better signal. We are sitting at the same spot in my house I have 4g he does not.
I'm in my basement have 4g, he does not.. if signal is important to you.. which probably is more than anything on the phone. my pick would be the s4.
Also to no, he did question switching to the s4.. and counted the days he had it.. only to find out he was a day too late.
Hope this helps some of you.. but as always to each his/her own.
happy phoning
EDIT
forgot to add that, there are also compatibility issues with games. Even though the phones have the same specs.. while the one is slightly underclocked to save battery. They should still handle the same games. Yet, the One can't download Vice City, because the phone isn't supported. It couldn't get Real Racing 3...
My experience with HTC was good at one point, till they started to forget about their phones after they put them out... I've had the Evo 3D, Evo 4G LTE, and I would talk about how crappy the plastic phones were that samsung put out.... was I wrong.