You are doing nothing wrong. You are using Google servers to store your contact info that syncs to your phone. If you want to keep your contact info without your Google account, do this. Sign into your Google account and let the contacts sync to your phone, go to the settings menu in the People app and choose export to storage, remove your Google account, then go back to the settings menu in the People app and choose import from storage.
Sent from my Nexus 4 using Tapatalk 4