Brother,
Im facing the same issue since 2 weeks, the battery also getting reduced dramatically on the same time.
Clearing contact cache didnt help. I removed most of my accounts like twitter , FB, IMO from account sync settings. But still not getting better.
If open contact app it take 5-10...