I've just dealt with this same issue, and I don't think the size of the Contacts Storage app/file has anything to do with how many contacts you have, deleting duplicates, etc. I think it's a function of having had an Android phone and a Google Contacts account for a long time, and that file just gets bigger & bigger.
I started having major trouble with Hangouts that I simply couldn't fix until I found a tip on the
Google Product Forums sites that fixed it. The tip was to clear cache and clear data in the Contacts Storage App within Settings. After I did this all of my contacts were deleted from my phone, and then they all re-appeared after my google account re-sync'd everything, which happened automatically.
I should mention that I make a point of saving all of my contacts within my Google account, not with Verizon's Backup Assistant or any of the other options that come up when you make a new contact. This meant that all I had to do was re-sync all of that contact data. I lost nothing.
After I did this my Contacts Storage went from something like 380 mb down to just under 20 mb, and everything on the phone has worked much more smoothly since, including the fact that it fixed the problem with Hangouts, which simply quit working all of a sudden and would barely function until I followed these steps.
Oh, and my phone is a Droid Maxx running KitKat.