I thought I'd post an update.
It turns out that the problem is isolated to specific speakers that I have. If I use other speakers or even if I plug in an audio extension cable to the "problem" speakers, the problem I was seeing goes away.
When look at the speaker connector, I can't tell the difference between it and the ones that work. These are powered speakers - maybe the speakers put a low voltage out on the speaker connector wire.
Strange.