Same here. I had a Moto X (2013, 1st Gen) that started having issues with this shortly after the upgrade from Kitkat to Lollipop (about 6 months ago). After taking a call or (now I understand, checking the Youmail app), any media audio output after that would route to the phone earpiece (rather than bluetooth or a wired headset or even the speaker), and there was no way to get it back to the desired output (that I could figure out.) I cleared the cache, wiped the phone, did a factory reset — nothing fixed the problem. A few weeks ago I bought a new phone, a Moto X Pure XT1575 with Marshmallow. Lo and behold, same behavior — no it wasn't something that was a screw up in an operating system upgrade (i.e. Kitkat to Lollipop on my previous phone.) I finally started to figure out it might be related to Youmail, because I (duh) realized it was happening after every time I checked voicemail on the Youmail app, or made a call using the call button on a voicemail. So I started searching online and thankfully found this thread. I was thinking it might be something related to Motorola tweaks to android, but seems to affect other phones as well.
Killing Youmail each time the behavior happens does 'fix' it until the next time I access the Youmail app. Thanks for that tip. Now at least I don't have to reboot my phone to get the media audio to work right.