I think there are 2 terms at play here:
Build QUALITY: how it is put together. Does it have large or inconsistant gaps between the parts? does it creak, flex, or anything like that? How does it hold up over time from day to day use? Polycarbonate (aka plastic) can "feel" cheaper but may (or may not) stand up to repeated drops better then metal. Makes me wonder what a full carbon fiber housing would do (not just the back, the whole phone). Quality is something that can be measured. take X number of phones and put them through various tests. Out of X phones, how many pass or fail that specific test?
Build FEEL: how does it feel in the hand? this is far more subjective. Metal has it's pros and cons, as does polycarbonate/plastic. So does wood, carbon fiber, kevlar, aluminum, stainless steel, iron, or whatever else you could possibly want to make a phone's body from. It's about finding the balance between cost, technical advantages/disadvantages, and what each manufacturer thinks their consumers want.
Personally (using my S4 as an example) I have it in a thin TPU case, for a few reasons. 1) without a case it's kinda slippery. the TPU offers more grip. 2) it offers slight impact protection, but more important to me keeps the screen and camera lens off the table, desk, or whatever I set it down on. Ideally I think a leather back would improve the grip factor a lot, but thin cheap leather (like you'd find on a phone back) won't wear that well plus I still need to keep the screen and camera lens off the desk. I also find the phone feels slightly light with this odd resonance (it feels hollow) when you pick it up without a case. Maybe a metal frame would improve this - but most likely the whole would need to be reworked.
I've held a S5 for a few moments. It still felt light, but I liked the back better. I don't recall it having that hollow/resonance issue but I could be wrong. The G3 also feels nice (good weight, but also somewhat slippery. Not as bad as the S4 though)
Personally I find I like the weight of metal, but it tends to be cold and slippery to me. I'd rather take a bit of a lighter phone if it means it's not as cold and slightly grippier.