1. TouchWiz is too "different" than stock android for my tastes. Sense 5.0 is pretty much the same interface only with blinkfeed as one of the panels (and you don't even have to make it your home screen). There are some differences with the menu structure and minor things, but I love the clean feel of it. Touchwiz feels like it permeates every corner of the interface. The sickly green color that Samsung likes to sprinkle everywhere in their interface is ugly as hell IMO.
2. AMOLED is still too punchy, to vibrant, too oversaturated, and whites still aren't "white". The HTC screen is incredible. Blacks are SUPER black - only barely less than AMOLED. The Samsung screen isn't even CLOSE to bright enough in direct sunlight. I can't believe I dealt with it on the Galaxy Nexus as long as I did!
3. The GS4 has quite a bit more lag all around. Not much else to say, but it felt like driving ICS on the Galaxy Nexus, somewhat.
4. I don't like that Samsung phones always have some kind of protrusion on the back for the camera. It just feels sloppy to me, and doesn't feel good in the hand.
5. There is absolutely no comparison on audio. The speakers on the One are about the only ones I've ever used on a top-billed phone that beat the iphone for loudness or clarity. The GS4 sports the same awful tiny speaker with compromised location that so many previous Samsungs have. It hasn't gotten better with age.