There are a number of programs to do this. As for settings I'm not exactly sure what settings to use. MP4 should work. Just make the file size smaller.
I'm sure someone else here knows what settings to use. If not go to other sections (EVO has one somewhere) to look for what they are doing. Just make sure whatever phone you're looking at has the same resolution as this one. I did a quick search and found THIS thread. I didn't read through the whole thing but it looks like a pretty good conversation about ripping movies for Android in general.
You need to use the baseline profile for HD main. The high profile will choke the Tegra 2 (or so I'm told, haven't tested it myself). The only file I have viewed on it so far was a 45 minute TV show. I used Handbrake and set it up for 720p, mp4(H.263) and set the total bitrate at 4000 and it played totally fine. Longer movies are going to be huge files with that setup though (45 minutes ran a little over 900 megs). I haven't had the phone that long though. I am going to try and play with some encodes this weekend and see if I can come up with workable settings.