The kernel, in any system, is the software layer between the hardware and the rest of the software. It contains the drivers and initializes the hardware, providing prioritized access for the rest of the software. You can make improvements to the kernel that make a big difference in hardware clock speed and voltage, how efficiently the hardware is used, and add support for new codecs, filesystems, etc used by a ROM.
A ROM, in this case, refers to the higher levels of the OS that you actually interact with. Its called that because its the other piece of software that's written to the read only memory NAND flash on the device, other than the kernel and baseband software. But changing the ROM allows you to change the look and feel of the OS on every level, change basic installed applications, change how different parts of the OS interact with one another, remove carrier and manufacturer bloat and launchers, and change to a more efficient filesystem, if your kernel supports it.
There's more, but those are the basics. Hope that helps you understand it a little better.