If you are running stock and unrooted I don't think you can remove it since its included as a part of the stock OS.
You would have to root your device and then manually delete/remove it using a root level access file explorer or Titanium backup to uninstall/delete the application.
Google Voice is a great app, I'd leave it and use it.
Use it for Voicemail instead of your carrier's, and get text transcriptions of your voice messages.