I have the same issue, started after updating to Lollipop
For the time being, I fix it by going into
Settings --> Apps -- > All
There will be three apss named Phone, I go into the second one and hit "Foce Stop".
Clears the icon, but it will reappear and get stuck if I reboot my phone.