Another component is that different carriers administer their phones differently.... Most carrier branded phones have been modified to suit the carriers needs... both hardware AND software. A Samsung Note 4 on Sprint will be different than one on AT&T.... different radios.. and a carrier like Verizon loads a fairly significant amount of added on software (i.e. Verizon Backup) whereas T-Mobile does not.
There was a study recently... I think they tested a Galaxy S4 across various carriers and the battery life (specifically screen on, active time) varied by as much as 20% across the different carriers, with the phones that have been most heavily modified suffering the largest drop in endurance.
Most all carrier branded phones get updates to the Android operating system from the carrier.... Depending on the phone and carrier, updates to your phone may take a long time, if at all, since the update has a long, long road... First, Google releases the new version, the OEM grabs the code base and modifies it to suit their needs. Then that is then sent to the carrier who them modify it again to match their guidelines. So even if the OEM decides to update your phone's base software, there is no guarantee that your carrier will accept it and pass it along to you.
The carrier/OEM relationship here is pretty complicated, and, in my opinion, completely unnecessary. Many of the carrier-based modifications are redundant, sometimes twice over.. with overlap from both OEM-based and Google sourced features. For every carrier feature that is quite useful (such as T-Mobile's WiFi Calling), there is one that makes little to no sense ($4.99/m VZ Navigator) It's a tangled mess sometimes. Many users get tired of this (including yours truly) and gravitate towards the Nexus line, which cuts out the middle men.
In a perfect world, carriers would stock unlocked, unbranded and unmodified phones (well, maybe T-Mobile can keep WFC.. that's legit

) but consumers vote with their wallets, and carrier-sourced phones are still the overwhelming majority of phones out there.