The actual "wiping" part (cache partition wipe from bootloader) really shouldn't vary in time depending on amount of data stored because it simply reformats the entire partition (at least, that is what it is CLAIMING it is doing). I imagine a 32GB model will take a lot longer than a 16. Haven't ever done this on the Nexus 5, but on the 32GB Nexus 10 it does take quite a while, perhaps 5 minutes or so?
Yeah, I've noticed that the cache wipe on the Nexus 5 takes considerably longer than on any of my other devices, like the 2012 and 2013 Nexus 7 and my old Razr Maxx. 10 minutes seems a little long, but I'm glad it finally finished.