I'm one of those guys who switched from a One to an S4. The reason why I still visit the One forums is simple: I actually liked the One better overall.
Here are my cons for both devices:
S4:
- Lag, lag, lag :-( I hate the performance - the constant stutter is just plain annoying
- Flimsy build quality (I constantly fear the screen might break)
One:
- Stupid menu button in many apps (especially annoying in the Kindle App, which happens to be the app I use most)
- Low-res camera with only semi-decent image quality (I don't actually use the camera very often - so this one's just a "yours is bigger than mine" type of thing, really)
- USB-port isn't in the center (device doesn't fit in my car kit)
I might still switch back to the One (soonish, as long as I still get a decent price for the S4) - we'll see. ATM I'm using my iPhone 5 again because of the S4's performance issues.