Really depends on your priorities and needs from your device. I own and use both the One and the S4.
If you value looks over durability and ergonomics, go with the One. It is sexy and shiny but it's button placement has been described as among the worst ever and I would have to agree. The S4 although plastic, has newer Gorilla glass and the plastic back allows for better durability and easier replacement. Also, if put a case on your phone the aluminum vs. plastic argument becomes moot.
If screen real estate is important, go with the S4 or G2. The One starts off with a smaller screen than those devices and then because it HTC decided to forego a menu key there is a large black bar across many popular apps such as Facebook, eating into the display. Allegedly this will be cured with an update but there are no hard release dates for US models and the phone has been out for months...which leads to another factor...
If timely updates are important, you may wish to look elsewhere. The One was not launched with the latest version of Jellybean and as of yet has not been updated in the US. Features like lockscreen widgets are as of yet elusive. Making matters more complicated, given HTC's implosion in the press each day(financial troubles, high level departures, espionage,etc.) the company's ability to provide updates is up in the air.
If you will use the photos for anything other than social media, look elsewhere as the lack of mega-pixels makes the One non-competitive with other flagships when it comes to details in pictures and the ability to zoom and crop. Further, look thru some of the threads here about the purple tint issue which apparently is so pervasive that HTC has promised a fix. That being said, Zoes and Highlights are unique and excellent gimmicks that haven't appeared on other devices.
If you keep your phone on vibrate and/or listen to media thru headphones, then BoomSound is negated and the One sounds no better than any other flagship.
If being able to modify the status light is important, the light on the One cannot be modified unlike on the G2 or S4.
There are many other differences but it really depends on your usage. The One is a great looker, rarely lags or crashes and has a good enough camera to Facebook etc. Hard to make a mistake with any of the flagships.
good luck.