Unfortunately, with only the stock app, you can't... they're randomly assigned when you add calendars.
However, there's a program in Google Play called "Set Calendar Colors" that allows you to select and change the color of each calendar. As to why the heck this isn't built already into the settings, is beyond me...