I finally figured out why it's doing this. It's certain phones, mines the Samsung LG Fuel. After a call of 1 minute and 30 seconds, no matter what phone app I use, the built in one, Hangouts, Talkatone etc... My EAR rubs on the screen just perfectly to hit the right buttons to switch screens...