I'm running encrypted after trying it on my N4 once 4.3 was released (both with stock rom and boot loader)
Personally I've had no issues at all with encryption and no slow down.
I chose to encrypt to protect the personal information I keep on my phone.
Note that to successfully encrypt you need a battery charge of more than 90% and the phone must be plugged in and charging (mains). If the charge is insufficient the phone will simply reboot as normal rather beginning to encrypt.
Sent from my Nexus 5 using Tapatalk