At this moment I have resorted to disabling Googles Contact App, to prevent it from getting contacts non stop, possibly draining my battery life at the background.
I have installed another simple contacts app, turning off all syncing, and it works fine. I can live without syncing no problem.