Under settings - accounts, you can turn off contact syncing for any accounts that you do not want contact info. This includes FB, twitter, etc. I only pull contacts from one place as it can be quite confusion as you state. Particularly when Android decides to join what it thinks are matched...