It's the location icon, if it has little bars next to it then apps will try to use your location, if it's just the "compass" icon then only 911 will. If you go into Settings > Location and change the setting, you'll notice the signal bars on the side change.
The GPS icon looks like a little satellite dish and only pops up when an app is actively using GPS.