I guess backup assistant had a use before true smart phones. You can disable backup assistant in settings by unchecking the box to sync your contacts. Join your duplicate contacts in the contacts app and you should be good to go.
FYI - the same type of issue can happen if your choose to sync your Facebook contacts. The contacts app will attempt to join exact duplicates, but it will miss some. For example, my wife uses her maiden name as a middle name on FB so old friends can find her. She showed up twice in my contacts so I joined them.
Sent from my iPad using Tapatalk