This link gives you an idea of the problem (and the solution)
https://carleisenstein.wordpress.com/2011/03/25/solved-contacts-storage-using-up-too-much-space-on-android/
I went from a 180MB size to 8MB size with following steps:
Stop synchronization from settings/accounts/google select account...