Hi Nilesh,
I am facing same issue. When did the problem started? For me this issue started suddenly so it can't be number of contacts. Neither I synced any other app at that time. I suspect it's due to Android update. Anyone else with nexus 5 facing same issue?