Welcome to Android Central! Does this persist after a reboot?
Are all of your contacts synced from Google, or do you have contacts synced with other accounts as well? And do you also have a lot of contacts that are only saved to the local Phone account?