I'm not disagreeing with you, but it's worth noting that when I was having problems with my previous phone, the Droid Incredible (which I got new), the replacement phone that I received (that was almost certainly factory refurbished) worked flawlessly. This experience has been described many times on this and other forums. Refurbished items are not necessarily items that were returned due to defects or problems--oftentimes, people just return something because they had second thoughts.
So personally, I wouldn't go as far as saying never buy refurbished. My advice would be to make sure there is a good warranty or return policy, whether it's new or refurbished.