The phone has weak radio. I have set here comparing with my wife's iPhone 4s. Her's is pulling 88 dBm, my nexus is pulling 100 dBm. She is getting almost double the speeds on the speed test app. I just wished I knew if it was software.
More than likely it is. The Charge had the exact same issues in the beginning.
Sent from my Galaxy Nexus