Buffering simply means that all the data that;'s downloaded has been shown and the app is waiting for more data.
Your 50mbps connection is between you and your internet provider. Between them and the Youtube server there may be a multi-gigabit speed available, or the path may be so crowded that you're getting 200kbps data transfer from Youtube to your provider. And there's nothing you or your internet provider can do about it, any more than you can't do more than 5mph on a 4 lane interstate during the rush hour when there's a 5 car pile-up closing 3-1/2 lanes up ahead.
The internet is like a highway in that respect - if there's a lot of traffic on a particular road (path), that traffic is going to crawl. The internet was developed as a means of getting data from point A to point B eventually, not immediately. (It was developed so that, in the aftermath of atomic war, when most of the nation's infrastructure was knocked out, the IRS would still be able to collect taxes from those places still having living people. If not this month, then next month.)
Internet 2, which we won't see for at least a decade, is designed for speed.
The internet now is to what the internet will be as spark transmission was to 3D HD widescreen TV - it's still in its infancy. About all we've done so far is used different coding to make faster speeds possible, and brought cable and/or fiber to the house, instead of to the phone company central office. It's like adding green paint and a flashlight to a Model A Ford.
Your normal fastest single connection is going to be close to 2mbps these days, so you can support 25 or more simultaneous full-speed connections - streams, downloads, etc., at once. But you can't make the data come down a slow connection to your provider faster. They can only deliver it as fast as they get it. (Which is why, for a household of 4, it's silly to waste money on more than 10mbps - that's 2 full speed connections each with 2 left over. As the internet itself gets faster [and it does, little by little - 3 years ago it was running about 1mbps], it pays to increase the bandwidth you buy.)