There is one way i know to fix this problem without factory reset. It is to go settings ->application manger -> all-> and clear data on contact and contact storage in app manager. This will fix the problem.
But... Your data for contacts will be wiped away. So remember to BACKUP!