Hopefully, you were saving your contacts to your Google (Gmail) account. If so this is not gonna be hard. If not you have a problem.
First, go to your contacts online and see if the contacts you want to save are in there. Don't worry about the 'extra' contacts from Yahoo.
This will restore them to any point they were at in the last 30 days. Just pick a day before the sync with Yahoo.
https://support.Google.com/mail/answer/1069522?hi=en
Second get all the contacts off your phone:
Settings > Accounts > Google > Select account > Disable contacts sync.
That will keep contacts on the phone from fouling up the rest of this.
Then:
Settings > Apps > Contacts storage > Force stop > Clear cache > Clear data
You should have no contacts on the phone.
Re-enable contacts sync to download the restored contacts list to the phone.
I f you have NOT been saving your contacts to Google, then probably all you can do is manually edit the list to eliminate unwanted contacts. If that's what happens I *strongly* suggest saving the corrected list to Google for easy restoration in the future.