I'm new here, I actually use a BlackBerry but I also have a G2 but I think I can help since this happened to me 2 days ago.
If you are using Google Voice switch to the carrier voicemail (settings - call settings - voicemail service) then call your phone from another phone and leave a message, listen to your message, then delete your message, your icon should go away. Then change back to Google Voice.
Do what the second guy said. I had the same issue with my G2 and my Blackberry back in the day. Simply call yourself from another number, leave a voicemail, check that voicemail, and it should go away after that.