Your phone and your Google account are two separate things. The only way someone would have access to your phone is if you unlocked it for them. Then, if you don't use an app lock, they can access your browser, and apps to access their accounts. If they aren't dependent on a password manager...