The first time rooting my phone i wanted to flash a custom Rom onto the phone because it would be cool to have a custom rom. So anyways when i tried to flash a custom rom either i did it wrong or the rom was faulty and it didnt work, it just went into boot loop. So i was like forget rooting my phone i just want to go back to stock so i went into recovery and where it said wipe i thought it would wipe everything and go back to normal. But nope it just wiped everything off my phone including the OS and I was so freaked out because my expensive device wasn't working. So that is where to root toolkit helped me bring everything back to stock lol.
Ah.... yeah... I think I know what happened.... Most custom ROMs only come with the OS and a kernel.... those are loaded to the /system and /boot partitions. When you flash the 'gapps' packages, those will also get loaded into /system as well. A few select ROMs include the google app stuff in their ROMs, but since these get updated often, most leave those out so a user can flash whatever google app packages they want. Typically, the /system partition is tightly restricted and really only low-level OS stuff gets write access. Users can only screw around in there with root access.
When you boot up your phone, log in your user, download apps from Google Play (updates to your system applications are also stored in /data... hence the option to "uninstall updates" when you look at the app setttings), configure your wallpapers and stuff... ALL that data gets stored in the /data partition. When you do a factory reset, it's pretty much just purging the /data partition. And since that is where all your customizations are kept, your phone is effectively reset to it's 'factory' state. Without all that configuration information, the phone goes into "First Boot" mode and you have to set all that stuff up again.
From what you described, the flash of the custom ROM failed... for whatever reason.... when you tried to boot, there was nothing there to boot TO, so it looped. The Factory Reset really did nothing here, other than re-format your /data partition (which you should have cleared when flashing a new ROM anyhow). NRT basically went back in and put all the pieces back together.. getting you back to square 1.
You're experience is not all that uncommon for people first trying out custom stuff on their phones. While the process is fairly straightforward, if the steps aren't followed correctly, things can go sideways REALLY fast.... and you end up in boot loop panic. Everyone's been there.... but the key is understanding what went wrong and how to get out of it.
Rule of thumb with the Nexus 5 is that if it turns on, you can fix pretty much anything.