the razr has a Kevlar back, now if the s4 had used Kevlar like the razr..that would have been really nice.
What I understand about construction (which I know a bit about) is that aluminum is a poor choice of material for external frames on electronic devices.
It's soft, and easily malleable unless you put a nice chunk of it.
Plastic on the other hand will shield internals far better than aluminum. The studies I've researched all pointed to high quality plastics with metal roll cages to be far better from a durability perspective.
The HTC One may feel sturdier but that's all it really is: feel. At the end of the day a drop at a bad place (screen) will result in damage for both phones while drops in more convenient places will yield slightly better results with the S4..