This icon appes when GPS is in use, which usally means some application or service running on background monitors the location. That icon should not apper all the time, or else you got app that constantly checking your location (like track recorder)
Considering you not running any app i mean some service is using it, Service is a part of application that runs on background because main application can't actually run on background.
You can check services running in Application settings in 3rd tab, you click application and you should see what services are on. Now you need to figure which one is using GPS, you can check it by turning them off one by one and see which one will make GPS disappear (thru i'm not sure), also try to avoid stoping services that sounds as a part of system since it might crash it, so check them as last.