Hangouts calling speakerphone only for Nexus 5?

[maxburn]

I'm using Google voice via hangouts and every time I never a call on my Nexus 5 it only works in speakerphone mode and won't switch to the earpiece.

Funny thing is the same app on my Xperia Z3 Compact works fine using the earpiece. It's rather annoying and pretty much the only reason I'm keeping the phone around. The phone doesn't have a SIM in it and it's doing WiFi only in the house.