Google writes an operating environment. It leaves it up to the manufacturers to design the hardware. So if one manufacturer uses port 5 to control brightness of the screen, and another one uses port 23, the software has to be changed for the device. (And there are thousands of those differences for each model phone.) Google doesn't (it should, IMO, but it doesn't) dictate how the hardware should be built. If Google licensed the manufacturers to only use Android on devices that ran on the software as-is, it would make for a much less splintered system, and once Google came out with an update, it would be much faster before the individual phone updates came out. (If one phone had a fingerprint reader and another one didn't, the manufacturer might want to take that code out for the phone that didn't, to allow more space for the user to use, for instance.) They could also not allow the carriers to modify the phones with bloat. (Apple does. If you're a carrier and want to sell iPhones, you sell the ones Apple produces. If you don't like the lack of bloat, don't sell iPhones [and be out of business in a few months].)
I don't think Google's crystal ball was working when they decided how to "do" Android. Then again, hindsight is always 20/20.
I mean why does each carrier have different 5.1 builds there is like 4 or 5 different 5.1 builds.
At least. There's at least one build for each model, and there may be different builds for the same model for different countries, or even for different carriers in the same country. 20 builds for one model isn't uncommon. And that manufacturer may have 5 different devices, so that's 100 builds for just one manufacturer.