If you have GPS turned on, and it's working (use
GPS Status & Toolbox to see if it is), it shouldn't be off by more than 10 meters (unless the atmosphere is really badly disturbed, then it can get as bad as 30 meters). Without GPS, if it's relying only on Wifi signals for location, it can be totally off - thousands of miles. If it's relying on cellphone towers, it
should at least be in the same general vicinity (if there's one tower covering a 10 mile circle, and you're not receiving any other tower, it's not going to be very accurate).
I'd bring it into a Device Support Center (ask for the nearest one at your local AT&T store - it might even be in that store) and see if they can figure out what the problem is. (If it's a hardware problem, and the phone isn't under warranty or extended service ("insurance"), they won't fix it for free, but at least you'll know.)