Well I'll give it a shot:
1. Metals feel better in the hand. I've held the HTC One and it's probabably the best feeling phone I've ever held. It's a joy to put in your hand. Feels super solid. I really doubt there'd be anyone out there (aside from plastic-fantastic Samsung fans) who would argue the HTC One feels worse than the S3/S4 in the hand.
2. Metal dissipates heat much better than plastic, which acts as an insulator, trapping heat. This is better for the internals as CPU's obviously heat up when under stress, so heat dissipation could play an important aspect of performance and longevity of the SoC.
3. If plastic was as great as you and others make it sound, why doesn't Samsung celebrate their plastic? On the S3 (and I believe S4), the band around the phone is plastic, yet it's coated with a metallic paint? Why try to dupe the consumer into thinking it's a premium metallic material? Seems like a slight-of-hand move by Samsung there. If plastic was that great, Samsung would have no reason to disguise the plastic with metallic paint.
4. Plastic is considerably cheaper to use in a phone build, yet the S4 will be priced just as high (or possibly even higher?) than competing high-end Android products. A polycarbonate plastic poured into a mold would be much cheaper than crafting billet aluminium CNC milled per device. Cheaper manufacturing costs for Samsung to reap higher profits, rather than provide their customers with the best experience.
5. With metal having a higher density and rigidity than plastic, it would hold up better in drops than the plastic would. Plastic would be much more prone to chips and cracks, where as metal builds would suffer from 'dings' rather than chunks being taken out, or the body completely cracked.
Issues like this wouldn't occur on solid-build metal bodies, but happen all the time on thin plastic builds.
Feel free to disagree with any of my points, but I'd much prefer Samsung went with a premium build for their phone. And I suspect all the Samsung apologists who are currently championing plastic would quickly changed their minds and welcome the premium materials with open arms if Samsung decided to make the S4 with that.