1. Thus far, my experience with both HTC and Samsung phones has generally been more favorable towards Samsung.
2. SDCard support. Unless the phone ships with 64GB (or at least 32GB), almost a requirement for me.
3. Camera. S4 camera is better, hands down. I don't think anyone will argue this.
4. Carrier. I'm currently on Verizon, which didn't offer the HTC One when I was shopping. I did weigh my options regarding switching carriers and/or waiting to buy a future phone, but in the end the One really wasn't a viable option for me.
Don't get me wrong, the HTC One looks like a great device...I wish I could have tested one out. It definitely has a few advantages (at least on paper) over the S4 as well, but the reasons above are why I own an S4 and not the HTC One.