Generally speaking if any of my phones are hosed up and I'm thinking about exchanging it then as part of my troubleshooting, before I call VZW or head into a store, I'm going to completely restore the thing to stock first just to rule out something I did.
If the phone's stock, unrooted and locked and it's still having an issue then it's their software or the hardware.
I've flat out told VZW before that I used the factory images and returned it to stock before I called and most of the reps will even skip their normal troubleshooting step of making me do another factory reset.
Same here, though I've never told them that I "returned it to stock", I just don't mention anything about ROMs, as admitting that can get your warranty voided. I do tell them, however, that I've already done a factory reset and the issue still persists. This, as you said, makes the Customer Service Rep immediately send you to Tech Support where they pretty much will start the process of getting a warranty phone (in my experiences.) Had to do this earlier this month to get my phone upgrade date moved up from Dec 29th so I could get the SGN.
But to the TS, they absolutely check. It voids warranty and it's really easy for them to spot.
A) Your bootloader will be unlocked, so when your phone is booted up, there's an "unlocked lock" icon underneath the "Google"
B) If you're rooted, you will have the Superuser application in the drawer, or root-only apps
C) Kernels, Baseband and ROM versions are described in the "About Phone" in the Settings menu.
I know they check, because my brother has a Droid 2 that has had a multitude of issues since the day he got it; I've gone with him each time to a store to get a new insurance/warranty phone (I'm the head of the account) and each and every time they check the "About Phone" section, for sure (the first time they tried calling him out on hacking it because the OS version was a mismatch, but it was because he hadn't updated it OTA, his phone was absolutely not hacked.)
Anyways
Unlocking the bootloader, rooting, roms, kernals and basebands... it's all relatively safe as it's very, very hard to brick the phone. It's also very easy to return back to 100% stock (there's a stock unrooted flashable ROM floating around and locking the bootloader is just as easy as unlocking it.) If you're anything like me, you will have all the tools and files set aside/on hand for when they're needed.