I did once, about a month ago. Everything crashed. My phone was basically unusable. Some people have done it with no problems, but not me. I guess it has to do with everyone's specific set up.
When I first got the phone, it seemed to run a bit faster and have a better battery life with ART. A couple firmware updates and many app updates later, the phone runs quite smoothly on Dalvik and I notice no difference with ART. ART also seems to cause problems with the camera app (among others) and won't let you do firmware updates. That being said, switching back and forth between the two is a nice way to reconfigure your apps if anything is causing strange behavior without a factory reset (for example, it fixed my problem with the phone dialpad not working with menus.)
I tried and it was a no go for me. Some people report that they are running ART with no issue(s). Good for them, but for some it's not quite prime time.