I think there is a market for both the iPad and the Xoom, as well as the Blackberry PlayBook. I've had Android phones in the past but now use an iPhone 4 on Verizon. While I love the size and battery life of my iPhone, I've had to jailbreak it to try to restore some of the functionality that is standard on Android and iOS is sorely lacking. I tried to like the iPad (twice, actually) but couldn't get over the fact that it is a bigger but slower version of my iPhone. It really doesn't seem like it's capable of doing anything that my phone can't.
As such, I think the only way that Apple could derail the Android tablets is if they released iOS 5 and it, too, was optimized for use on tablets and provided functionality beyond what the iPhone can do. To be honest, unless iOS 5 is a huge step forward, I just may go back to an Android phone when I'm eligible for an upgrade in July as I paid full retail for the iPhone.