I have Samsung Galaxy S4.
Here is one way I know to fix this problem without factory reset.
It is to go settings ->application manger -> all-> and clear data on contact and contact storage in app manager. This will fix the problem. But... Your data for contacts will be wiped away. So remember...