Turning off speaker when not on a call

[haldroidx]

Sometimes I have a speaker icon in my notification bar and, when someone calls me, I find that when I answer, the phone begins on speaker mode, which I don't want! It's only recently that this started happening, maybe since the last Android upgrade. How can I keep the phone from starting in speakerphone mode when I initiate or answer a call? Failing an answer for that, how can I toggle speakerphone mode off when I'm not already on a call? I can't find a way to do that.