Awesome advice, it solved my problem!
To recap: My mic sounded horrible, delivering only hellish scratchy noises to the other side. I could always hear the other side. The horrible sound from my side went away when I muted my mic, and the audio worked fine on speaker mode.
Following your...