Are the addresses synced from an account, like your Google account? Be careful, because if they're synced and if you delete the contact on your phone, then you'll end up deleting those contacts from the source.
If you were to go to Settings/Apps/All and select Contact Storage, and then Clear Data, that should clear out the contacts, but if your Contacts are set to sync with Google or Yahoo or whatever (in Settings/Accounts), then the Contacts app will just repopulate again with the next sync.
Is there any easy way clear the data out of this app?
I might use it but it's loaded about 400 addresses that I don't use and it's a pain deleting them 1 by 1.
No, your stock contacts app will not be deleted. If you're not rooted (which is like jailbreaking an iPhone), you can't uninstall the stock system apps that came preinstalled. You can, however, disable the stock Contacts app by going to System Settings/Apps/All (or the App Manager on Samsung devices), selecting Contacts, and tapping Disable. This will prevent the Contacts app from opening, which might clear some system RAM to allow other apps to reside in active memory. But I'm not sure if disabling the stock Contacts app might interfere with other functions, like searching for Contacts from the phone Dialer.