Depends on how it's coded. It could be included in the install image, or it could be downloaded as part of the installation procedure. Both are possible.
But yes, when you delete a factory installed app you're only deleting it from the installed version of the OS, not from the permanent ROM...