What device do you have?
If you just don't make a lot of phone calls, you might see that. I have a Moto X Pure Edition, and I don't make many phone calls, so just because I called my eye doctor's office a couple times to schedule an appointment and to pick up my contact lenses, the eye doctor ended up in frequent contacts. It doesn't bother me, since I rarely make phone calls anyway, but I could remove the eye doctor from frequent contacts if I wanted to do so.