Voicemail notifications are tied to your carrier, so you're gonna have to check in their voicemail app or calling in to make sure no new voicemails are waiting there.
Dial into your voicemail, double check there is nothing there. Once the automated voice says, "there are no voice messages," close the dialer by pressing the * key, and let the voicemail service close on its own.