In the settings for GoSMS you can disable how it matches contacts. I believe it has options to match them by name, phone number, and email. The more you check, the slower it loads. I believe the fastest implementation is matching by phone number.
I currently have it matching by phone number only.... It still refreshes each avatar as I scroll or exit/enter the app. Just curious, how many threads do you have in the application?