I think I figured it out, mostly. Gmail apparently by default adds "other contacts" that you reach out to. I turned off that option on the Gmail Settings General tab, moved any of the "Other Contacts" I wanted into "My Contacts", deleted the remainder, and my phone and Google Account kept up...