No, you're not alone!
Both my wife and I have experienced this on a nearly continuous basis since we got the HTC One. The volume goes up and down a few times after a call is made and then usually stays so low you can't hear the other person. This seems to be a problem only with the combination...