I've been having random restarts a lot lately also. Sounds similar to you, there is no pattern or 1 app I am using that I know of, causing it. I came here today to see if anyone else was having this issue, and I was wondering if there was a log to look at or something that could tell me if a specific app was causing it. I love this phone but I am at the point now where I hate it. This morning alone I had 4 restarts. I got the phone on launch day, and maybe had 1 restart every couple months or so. Then after the first update to I think 4.0.4, it started happening a little more, but not more than maybe 1 time a month. Now since updating to 4.1.1 it happens everyday at least once, but more often a couple times a day. Today was it, I am going back to where I got the phone and see if they can replace it or I can get a different phone. I did buy some insurance plan, so I hope it will help me. I will check out the logcat apps first.
Just to add... when an app crashes, I get the message about sending a report, but it will say what app caused the problem. This is different, the phone just goes black, and the power/wake-up button does nothing. Initially I was pulling the battery when that happened cause I thought the phone was not responding, but I later realized its restarting and after a few seconds I get the nexus X on the screen as its starting up. Or if I am not using the phone and it's sitting on my desk at work or something, I will see the nexus X appear and watch the phone start up.