I inherited a Nexus 4 two weeks ago.
My only experience with smartphones prior to this was with windows phone. I had a Lumia 710, 822, and 925. I still have my 925. It's been my daily driver for over a year now. I love it. It's snappy and responsive and the cameras it takes are amazing. Because I haven't had experience with other smart phones other than lower budget windows phones, I never saw an issue with my Lumia 925.
So I've been running with the Nexus 4 as my intro device for about a two-ish weeks. Half of the time, though, was spent without a sim card so I only used it for the apps/ecosystem while my Lumia was used for calls and texts (This was due to inheriting a cyanogenmod ROMed Nexus 4 and it having bugs with sound over calls).
Eventually I got fed up with carrying two phones and just looked up how-to guides on unrooting/reflashing the stock Nexus 4 os. So no more Cyanogen mucking up the works.
They are both great phones. If you have a Nexus 4 without any non-Google roms, I can't see why you would have any difficulty figuring things out. And the learning curve people seem to warn about is probably aimed more towards iPhone users, I would guess. It seems like Apple Products in general just breed dumber users who can't noodle around briefly to figure out how to do things. I had no issue figuring things out.
There are a few annoying things that I miss from WP8.1 that I cannot seem to replicate on my Nexus 4 via setting menu spelunking or through apps.
The biggest annoyance is that you can't edit and silence notifications from applications through any menu, like you can in WP8. You have to actually open each individual app and turn off notifications, or notification sounds... And sometimes you can't just turn off the sound, so you have to turn it off completely.
Another thing I miss from WP8 is the people hub. The default contacts app on android does not, for whatever reason, have the ability to quickly jump to specific letters of the alphabet to filter our contacts. You have to manually type in "m" to bring up the M names, instead of tapping a letter and then choosing a letter from the alphabet screen. I don't know if I made any sense for this point, but you can't sort by letter nearly as efficiently on Android.
Make sure you look into widgets. You will miss live tiles. You can fill the void with some good widgets. Weather widgets and calendar widgets.
Also if you're coming from Windows Phone, Google refuses to play nice with Microsoft - so if you use Microsoft's services (Outlook/Live mail, Onedrive, Bing Rewards, etc...) You are gonna have to deal with google ******** at you whenever you try to set those services default instead of theirs.
Also, I don't know the reason for your desire to switch, but I know lots of people seem to rag on Windows Phone for having no app support.
Now that I'm playing on my Nexus 4 I find myself opening Google Play Store, scrolling through apps, and wondering what I thought I was missing. There's nothing interesting or new that you can't get on WP8, save for Snapchat, and major business/corporation apps. I thought it was going to be an oasis of awesomeness that was gonna open my eyes to some whole new world of Apps... It's not. Though Facebook's android app is better than the one on WP8.
But overall I think the Nexus 4 is as simple as you can get on Android. Though it is quite slippery.