I tested this a bit last night using Handbrake. It can definitely be done and it?s not too hard. So far all my playback on the Xoom has been with Rockplayer lite.
Personally, I am a big believer in re-encoding videos to best optimize playback on a portable device, even if they can natively handle the original file. Therefore, I?ll always convert the file to 1280xXXX progressive output. That said, it kinda depends on what source you are starting out with.
Handbrake?s de-interlacing is not the best, so if you are converting a recorded 1080i broadcast (Tv show) you?ll likely get some de-interlacing artifacts (I?m debating about playing with some other options to achieve better deinterlacing.)
For converting Blu ray rips, I?ve so far just left the framerate at the native 24p and the Xoom plays them fine, although you get the usual judder that occurs when playing 24p material back at 30fps. I still need to play around and see how well the handbrake handles telecineing the material. It may provide a little better results.
In any case, set the output resolution to 1280 by 720 for 16x9 material, or set it to 1280 by 544 for cinemascope material but be aware that BRs are encoded 16x9 (non-anamorphic) so adjust the cropping if it doesn?t automatically pick it up the black bars on the top and bottom.
Set the encoder settings to 3 B frames and turn all the other options off. For target bitrate, so far 1GB per hour (for fullscreen 16x9 content) gives a decent tradeoff of quality to size, imo. You?ll still see some blocking in fast motion of scenes with fog/fire, but otherwise it is acceptable. Increase if you have space to spare and want better image quality. Obviously, cinemascope material can get away with a lower bitrate as the image on the screen is smaller.
One last note, if you are converting DVDs, don?t be too surprised to find they look less than stellar on the Xoom. DVDs have a native resolution of 720x480 to begin with (I wouldn?t bother encoding it at anything other than the native res.) So don?t be surprised when up scaling at playback to 1280x720 causes it to look soft. It?s not the Xoom?s fault as the source material you start with isn?t good enough to watch at that resolution, held a foot or two from your face.
-Suntan