General ROM FAQ

[Jerry Hildenbrand]

Appreciate any and all input (and proofreading!). This is a work in progress and subject to change

Begin------------------------
Android ROM FAQ

1. What is a ROM?
A. Parts of a ROM
B. How is a ROM packaged?
2. How do I install a ROM.
A. Got Root?
B. Recovery
C. Nandroid
D. Copy and Flash
3. What if something went wrong?
A. Reflash the ROM
B. Flash a different ROM
C. Look for an official recovery tool
D. "Borrow" a Nandroid backup
E. A trip to the store
4. How do I uninstall a ROM.
5. Building your own ROM.
6. Conclusion
7. Disclaimer


1. What is a ROM?
If you?re new to the Android platform (don?t be bashful, all of us were not too long ago), you?ll see terms floating around the forums that you might not be familiar with. ?ROM? is one of those terms. A ROM image is a data file that contains information used on a Read Only Memory chip. For our purposes, that means a complete system image of an Android device. Each Android device has it?s own ROM image that contains files and code needed to boot the device up and run Android on it. But this is only part of a ROM. A ROM also contains a GUI (graphical user interface), required and useful applications, support files for those applications and the kernel. Let?s have a look at the parts that make a ROM.

A. Parts of a ROM
i. The kernel.
Android (like many other Smartphone operating systems) runs on the Linux kernel. The Linux kernel was created in the early 1990?s by a gentleman named Linus Torvalds in Helsinki Finland. It?s incredibly stable, incredibly friendly, and incredibly difficult for the layman to understand and modify. Thankfully it?s also very popular so it has been ported on to a multitude of hardware, including our Android devices.

Think of the kernel as an interface layer between the hardware and software on your device. The kernel decides when things happen, such as the LED indicator gets lit. An application sends a request to the operating system to blink the LED. The operating system then sends the request to the kernel, which makes the light flash for the amount of time requested by the OS.

What sounds like a round-about way to get things done is also what makes the system so scalable and robust. Application developers only have to code in a way the operating system understands and the kernel makes it work on the hardware. This also keeps the application running in it?s own user-space and separate from the kernel. That means when you run the latest uber-cool app that wasn?t designed for your particular OS version, or is still very beta and it crashes, the kernel gives you the option to Force Close the application and the kernel can run untouched.

In a standard Android ROM (we will leave developer images and the like for another discussion) the kernel is bundled along with a set of instructions that tell the device how to load the kernel and the OS during boot. This is the boot.img that you see inside a zipped ROM that your not able to easily open. The device knows to extract this image to internal memory (the ramdisk) and follow a series of scripts (init scripts) to load the kernel and then the other portions of the OS. That?s what?s happening while you?re watching the boot animation. Interestingly enough this is done the same way for a PC, your smartphone, an Android tablet, or even a smart Linux powered toaster. If you?re feeling exceptionally geeky, plug your Android phone into the USB port on your PC and let the PC boot from the USB device. No, it doesn?t actually load, but you can watch the animation while it tries to match up the hardware support with what?s inside your PC. As I said, Linux is amazingly scalable and as a result so is Android.

ii. The operating system.
Once the kernel is loaded, the init scripts tell the Operating System to load. Android is the user interface for a custom built Java virtual machine called Dalvik. Dalvik was written by Dan Bornstein, who named it after the fishing village of Dalvik in Iceland, where his family originated from. The debate of which Java VM is superior is best left for another discussion, so I?ll simply say that DalvikVM is a register-based machine versus true JavaVMs which are stack based.

The Dalvik machine creates executable files (.dex files) which can be interpreted by the OS and run by the end user. These .dex files are OS version dependant. That simply means that applications and core functions built to work with one version of Android may or may not work well with other versions. Google provides the tools through it?s Software Development Kit (SDK) for applications to communicate with the OS.

iii. Core functions.
No smartphone would be complete without a set of functions that allow the device to be used as intended. Things like the phone and dialer interface, the calendar, the messaging system are core functions of the Operating System. In Android, these are run on top of the kernel as separate applications. The merits (or lack of) of providing these needed functions as separate applications is once again best left for another discussion, but this is what allows developers like HTC or Motorola to replace the standard functions with alternatives that provide a different look and feel from stock. HTC?s onscreen keyboard or Motorola?s MotoBlur contact list are great examples of this. The ?little guy? isn?t left out of the mix either. Handcent SMS or Chomp SMS can integrate into the OS very well, as most of us already know.

An additional set of Core Functions are provided by Google. Popularly called GoogleBits, things like Gmail, sync, Gtalk and the Android Market are applications written by Google that give an extra set of useful functions to the OS. You?ll find these on all smartphones, as well as many other Android devices.


iv. Optional applications.
These are applications provided by the manufacturer to give the device even more usability. Things like the Amazon MP3 store, PDF readers, Corporate Calendar etc. allow you to do even more with your device. Remember - Droid Does

 

[Jerry Hildenbrand]

B. How is a ROM packaged?
In most cases a ROM will come packaged in a .zip file. The recovery image’s kernel (yes, it has one too!) has the ability to unzip and copy the contents into the correct place. Inside this zip file is a folder (META-INF\com\google\android\) that contains a script prepared by the ROM “cooker” (another of those techie terms - it means the person(s) who developed the ROM) that tells the system what to format, what to copy and where, and any file operations that need to be done. Each device does things a bit differently, but this script is where it all gets done. More on this folder later.

You’ll also see a /system folder. This is the meat of the ROM. It has the necessary OS files, the Core functions, and any optional applications the cooker decided to include. The folder is structured the same way it is on your device - /system/app, /system/framework, etc. The whole tree is usually copied over and the existing /system folder is overwritten. The cooker uses the script to tell the kernel to erase the existing system folder, copy the new folder over, and set the file permissions.


Sometimes you will also see a data folder. This usually is space set up for optional applications, including optional system tools like busybox or SuperUser white list. These applications could be placed in the /system folder, but placing them in the data folder makes it easier for the end user (you and I) to remove or update them as needed.


You’ll also notice a META-INF folder. This contains the update script we talked about earlier, as well as secure keys that need to be provided so the device knows the update can be trusted. A special note needs made here. Trusted means that the update is trusted to be in the correct form to load the device. It in no way means the ROM is safe from malicious code. Anyone is able to use a set of test keys and create a ROM that will flash and run your device - even those people with bad intentions. Flashing and running a custom 3rd party ROM is putting faith in the cooker that he or she not only knows what they are doing, but are honest as well. Also, some Motorola custom ROMs will have a small update.zip stored inside this folder to be run on first boot of the device.

Finally we are left with the boot.img file. This is the kernel and ramdisk image we discussed earlier. Your phone copies this over to be decompressed and run when the device boots.

 

[Jerry Hildenbrand]

2. How do I install a ROM?
In this section we’re discussing how to install a custom 3rd party ROM. ROMs from the manufacturer usually have a utility that runs on your PC to flash and load the new image.

A. Got Root?
Custom ROM’s simply will not load on devices that aren’t rooted. In theory, it may be possible to sign a 3rd party ROM with the keys that the stock recovery image will flash, but for the most part you need to have flashed a custom recovery image before you can change your device’s ROM. Instructions and tutorials on how to root your device are all over the internet. Some are good, some are bad. The hacking forum is a great place to go and learn more about rooting and how to successfully get it done on your device.

B. Recovery
Most Android devices have had a custom recovery image written for them. This will overwrite the stock recovery image, allowing you to flash 3rd party ROMs as well as giving extra functionality. Help with finding and flashing the custom recovery image for your device can also be found in the hacking forum. The installation of a custom recovery image also allows for a very important function. Backup and restore.

C. Nandroid
Nandroid is a set of bash scripts and code written by Brainaid and infernix @xda-developers that copies the state of your system and stores it in a folder on your SD card. You can then use the restore function of Nandroid to restore to this point at any time. This is a priceless feature and reason enough to root your phone. It’s included by default in most custom recovery images, and the code is freely available to use if you’re inclined to write your own recovery image.

In most situations, using Nandroid to back everything up is easy:
1. Verify you have a memory card with enough free space (~300MB to backup, ~500MB to restore).
2. Reboot your device into recovery. It’s slightly different for each device, once again hacking forum FTW!
3. Navigate through the menu and select the Nandroid Backup function.
4. Apply your choice and wait for the device to tell you it’s finished.

It’s always good practice to copy the entire nandroid folder from your SD card to a safe place. You can then copy it back to the SD card if the card is ever damaged, lost or erased.

D. Copy and Flash
You’re rooted, have downloaded a custom ROM, have your system backed up and are now ready to flash your device. This is not nearly as scary as it sounds.

1. Mount your SD card to your PC, and copy the .zip file to the root folder of the card. Don’t unzip the file, and don’t look for a folder called root. The root folder in this case means the base folder, what you will see when you mount your card to a PC or the device.
2. Reboot your phone into recovery.
3. Navigate through the recovery menu and select the flash update option. Depending on your recovery image, the file may need to be named update.zip, or you may be able to select any zip file on your card as long as it’s the correct format. The cooker knows this as well and if the ROM needs to be named update.zip it will be.
4. Apply your choice and wait for your device to tell you it’s finished.
5. Reboot.

It’s worth noting that many times a new ROM will require that you wipe and factory reset your devices data. While inconvenient, it’s often necessary to get rid of the old data as it may be incompatible. As long as you’re using the cloud for calendar and contacts, they will be re- downloaded and stored back on your device automatically.

 

[Jerry Hildenbrand]

3. What if something went wrong?
There?s a lot of data transfer going on when you flash a custom ROM, and the steps to follow are pretty specific. Things can and sometimes do go wrong. If you followed the advice about backing your system up, you?re thankful you did as you reload your backup. If you didn?t, not all is lost. here?s a few things you can try to revive your bricked device

A. Reflash the ROM
If you?re sure the ROM you have downloaded is built for your device, and is in the correct format, try flashing it again. Computers and smart electronics are touchy, and any interference while the bits and bytes are copying can lead to failure. If in doubt, download the ROM again and be sure your cables are in good shape. I can tell you from experience that dogs AND parakeets can be bad for data cables . Don?t have multitudes of things running on your computer while you copy the ROM to your SD card, and be sure not to disturb the device until the flashing is complete.

B. Flash a different ROM
We?re all human, and all make mistakes. Maybe the cooker uploaded the wrong file, or you downloaded a test build that wasn?t ready for flashing. A quick search or a post on the forums should lead you to a known good ROM for your device. Find it, download it, copy it to your SD card. At this point it?s always a good idea to wipe your device before you try flashing. There?s a good chance the data is corrupt anyway, so a clean slate is always your best bet.

C. Look for an official recovery tool
Many devices have had the official recovery tools leaked to the internet. A quick search for Verizon Droid Eris RUU will quickly take you to that units recovery tool from HTC. Other models have had pure stock images made by the community for them. The Verizon Motorola Droid is a great example of this. A ROM that is a clone of the factory software is available and will put things as they were before you ever started hacking your Droid. If you need help finding these tools, post in the hacking forum for assistance. Make sure your post title is descriptive, ?Help finding MyTouch 3g official recovery? or ?need help flashing Pulse back to stock? will probably get the attention of someone who has already solved the issue you?re having. There?s also a troop of helpful people that know where to look for such things, and your post may draw their attention as well.

D. ?Borrow? a Nandroid backup
Some people are clever. They had the foresight to pull a backup of their system before they entered in their personal data. Most times, this can be passed to another user with the same device and be used to get your system back where it was before the problems started. I encourage everyone to back up a copy of a bare system that hasn?t been signed in to Google next time they have to wipe their data. This is a very easy way to contribute to the community and help your fellow Android addict when they?re down.

E. A trip to the store
Sometimes nothing helps. While rare, it can happen to anyone. Your carriers repair center has access to all the tools to flash a phone back to stock. This should be your last resort, as it?s not your carriers fault that you decided to circumvent their software protection and customize your device. Always try to find a branch office of your carrier with repair technicians on site. It?s not fair to swap out your device for a replacement and pass that expense along to everyone else. You were warned when you rooted your device that it may potentially void the warranty, so be prepared for issues if you?re honest with the technician. It?s been my experience that honesty is always the best policy, and if asked about any rooting or changing of the software you should just tell the truth. Chances are the tech won?t even ask, and will simply hook the phone up to their computers and attempt the recovery. If they do ask, that?s a sign they are savvy enough to check and find out the truth anyway, so being dishonest serves little purpose. Besides, what kind of tech hasn?t already rooted and Rom?d their Android phone anyway ?J I know the tech in my local store runs MoDaCo?s ROM with Metamorph themes on his Hero, because he proudly shows it off to his more geeky customers!

 

[Jerry Hildenbrand]

4. How Do I Uninstall A ROM?
Short answer - you don?t. Flashing the memory of the device erases all the old data, so simply uninstalling leaves you with nothing. What you need to do is reflash the device back to it?s factory state. You can restore the device?s factory OS by flashing a copy of the Stock ROM. This will keep your device rooted, and keeps any custom recovery image you have flashed intact. This would be my preferred solution, as you still have access to the backup and restore functions of your recovery image, and are better able to maintain your device with Superuser access.

If for some reason you need to get everything back like it was when you bought the device, you?re going to need a copy of the manufacturers recovery update tool. As mentioned above, many devices have had these leaked on the internet. If you device isn?t one of these, a trip to the store is in order. Please don?t be deceitful and sabotage the OS to get your local tech to wipe and flash. Tell him/her that you tried some of the tricks you found on the internet and are worried once you realized that it was no longer ?official?. Most times a technician will be happy to get you back to compliance. In reality, it?s so easy there shouldn?t be any issues. Simply plug in the phone to the store?s computer and click one icon. Remember - technicians are people. They appreciate honesty and aren?t very happy when they feel they are being deceived.

 

[Jerry Hildenbrand]

5. Building your own ROM
Since we know a little bit about the internals of a ROM, and how to install them, the next logical step is building our own. ROM cookers try hard to make something that works for the majority, but everyone's needs are different. A few things to think about before you get started.

A. Do I need to do the extra work of cooking my own ROM?
Unless you have very specific needs, chances are that it would be easier to flash a pre-cooked custom ROM and swap a few things rather than cook your own. Applications and scripts can be added or removed to further customize the work already done. If you just want to add the 2.1 Gallery to your Droid (for example) it's easier to add it to what you are running than it is cooking your own. Of course if you're like me (Just admit it. The laughing only stings for a little while ) you'll have fun building your own. Read on.

B. What base to build from?
So many choices. OS version is always the first consideration. No point in trying to build a MotoBlur ROM then deciding to use Android 2.1, because it isn't ported yet. As beginners, it's best to leave the version porting of complicated things like OpenGL or Home replacements to the pros. Once you decide what version of the OS to use, then you need to consider the interface. At the moment, 1.5 and 1.6 ROM's can be built with HTC Sense or Motorola MotoBlur. 2.0 and 2.1 ROM's can be built with Vanilla, or with the extra eye-candy from the Nexus One. Things change fast, and eventually you will be able to mix and match an UI to any OS version thanks to the open source nature of Android and the strong development community.

C. Do you plan on releasing the ROM to the public?
I can't stress how important it is to share your open source work. Any ROM you build has the potential to be the next darling of the community. Remember, there will always be people who aren't able to cook their own ROM, and maybe yours will be a perfect fit. If you do decide to share your work, you will need to be prepared to support it. Some people will have issues, and look to you as a developer to help them get them sorted out. It's also a good idea to credit others if you've used their work, and to be prepared to release the source of anything you've modified so the next user can make it even better,

D. Can I use Windows?
The good and short answer is YES. Most of the process can be done using Windows provided you set your system up and get a few tools.
1.Show hidden files and file extensions. This can be done in the folder properties, and can be set to be the default when using explorer. Different versions of Windows have different methods, so Google the instructions to do this for your version of Windows.
2.Graphics editors. You need one, and it needs to be a pretty good one . I recommend Gimp, or if you have the resources, Photoshop. Any graphics editor that supports .png images and has automation will work.
3.A decent editor. Notepad just won't cut it for this. Download Notepad++ to edit and create scripts in a readable format.
4.7zip. WinRar and WinZip are nice applications, but 7zip is needed for some of the things we will be doing.
5.Other tools needed will be listed in their respective sections.

Once you've decided what you're going to build, it's time to get started. Previously we went over the parts of a ROM that make the whole. The best place to get those parts is from a pre-existing ROM. You do have the ability to pull everything off your phone, but this is work someone else has already done and decided to share. No need to re-invent the wheel. Download a stock version of the ROM you would like to build, unzip it to a working directory and have a look through it's contents.

E. The Boot.img
We already know that this is the kernel and ramdisk needed to boot up the device. You can pull this apart and swap out the kernel if you would like. For this, you're going to have to run Linux. If you're not able to run Linux natively, download Sun's Virtual Box and an easy to use Linux distro like Ubuntu or Fedora. We will leave custom kernel compilation for another day. To extract and rebuild the Boot.img we need some tools. These are in the attached Boot-tools.zip file found in the first post. Extract the entire zip file into a tools directory in the top level of your working directory. Open your terminal and navigate to your working directory, then enter the following commands:

/tools/extract-kernel.pl boot.img
/tools/extract-ramdisk.pl boot.img

This will place a copy of the kernel and ramdisk in your working directory. Overwrite the kernel image with the kernel you would like to use. Then re-package everything as so:

mkbootfs boot.img-ramdisk | gzip > ramdisk-boot
mkbootimg --kernel boot.img-kernel --ramdisk ramdisk-boot --cmdline "no_console_suspend=1 console=null" -o newBoot.img --base 0x19200000

Voila. You now have a new boot.img using the kernel of your choice.

F. Apk file editing
Apk files are the instruction to install and run applications in Android. When they are accessed, they create an executable .dex file that the JVM can read and run. Editing the contents of an apk to change the look of an application is pretty straightforward. Create a separate working folder for the apk file. Open the original apk file with your archiver. Extract the contents to your working directory. Carefully look at the contents of the /res folder. This is where the graphics of the program are held. These can be edited, with a few restrictions.
I.Edit the original file rather than create a new file.
II.Don't change the dimensions or resolution of the original.
III.Don't attempt to edit files that end in 9.png
Look at the application as it's running on your device, and search through the apk files to determine what element to change. This is all trial and error. Once you've made your edits, copy the edited files back inside the apk file using your archiver. Push the edited apk to your device, reboot and test. Once you're satisfied with the look and feel, copy the modified apk to the correct location. More on the “correct” location later.

G. Folder layout
The OS needs things laid out in certain expected folders. There is some leeway, but not a lot. In general:
I.Modifications to the framework need to be in /system/framework.
II.System applications and Core functions need to be in /system/app.
III.Scripts and other executable items need to be in /system/bin or /system/xbin.
IV.The META-INF folder is off limits for anything not associated with the recovery image.
Some examples –
The Google Maps apk usually is installed into /system/app. This works pretty flawlessly, and makes for easy updates provided you update directly from the Market. The only drawback is that the /system/ directory is mounted as executable and readable but not writable. This makes perfect sense, as we don't want anyone changing things in the /system without prior approval. Users stuck on 1.5 (ahem HTC Samsung and Huawei I'm calling you out ) no longer have Google maps in the market. Any future development of the Google maps application will be user driven, so most custom ROMs have placed the maps.apk in /data/app. This way the end user can easily overwrite it with a newer version.

It's nice to have a shortcut to a long command that may need to be entered via a terminal on the device itself. A cooker can write a Bash script and place it in /system/xbin so it's in the end users path. This allows the user to type in a single word command to preform a long series of complicated commands.

 

[Jerry Hildenbrand]

H. META-INF.
This folder gets it's own section, as it has pretty specific rules. Once you open it in your working directory you will see it's pretty bare. We went over it's contents earlier, let's get a bit more in depth. com/google/android/ contains the update-script only. It can never contain anything else. Here's a basic update-script that we can break down and see some of it's functions.

show_progress 0.1 0
copy_dir PACKAGE:system SYSTEM:
set_perm 0 0 04755 SYSTEM:bin/su
show_progress 0.1 10

show_progress 0.2 0
write_raw_image PACKAGE:boot.img BOOT:
show_progress 0.2 10

Line 1 - shows a progress bar, and sets it to the beginning
Line 2 - tells the recovery image to copy the system folder of the ROM to the device, overwriting the current /system folder
Line 3 – sets the permissions on the file /system/bin/su
Line 4 – the progress bar should be finished
Line 5 – starts a second progress bar
Line 6 – tells the recovery image to copy bit for bit the ROMs boot.img to the device as the standard boot.img
Line 7 – tells the progress bar to be finished

As you can see, this script is very short and sweet. Have a look at a complete custom ROM's update-script to see how much you can actually do with one. Once you get into a more complicated script you'll realize all the commands are a variation of the above.

Your update-script is a plain text file that can't be written with the standard Windows editors. You will need to use the Notepad ++ application to create or edit one, and be sure to save it with no file extension if you're using Windows to cook.

In your working folder you'll also see other files in the META-INF folder. The files CERT.RSA, CERT.SF, and MANIFEST.SF are the keys used to sign the file. We'll talk more about packaging and signing later, but for now these files need deleted. The keys are no longer valid when you repackage the ROM and new ones will be generated.

I. Signing.
Once you have the ROM set up the way you like it, you need to package and sign it for delivery to the device. This is commonly known as an update.zip and will allow the recovery image to flash your ROM through the recovery menu. This is done in two steps.

1.Packaging
When you opened the original ROM you noticed everything was very tidy and packed into subfolders. We need to replicate that. Use your archiver and create an empty .zip file. Use the name of your ROM.zip or update.zip if your recovery image requires it be named that way. Open this archive and drag your top level folders into it. System, data, META-INF and boot.img should all be in the root of the .zip file and not nested inside your working folder. Once you're confidant you have the file structured correctly, save and close it. You now have a ROM image ready for signing.

2.Signing in Linux
Attached to the first post is a file called LinSign.zip. Download and extract the file into your Android/SDK/tools folder. Optionally you may add this to your path in your ~/.bashrc file

export PATH=$PATH: /android-sdk-directory/tools/sign/

Open the terminal and navigate to the directory that holds your completed ROM.zip file. To sign enter the following:

java -classpath Android-SDK directory/tools/sign/testsign.jar testsign NAME-OF-ROM.zip NAME-OF-ROM-signed.zip

This will create a signed copy of your ROM, inject the rsa keys to allow flashing and place it in the folder alongside your original ROM.zip. The new file is ready to copy to the SD card for flashing and testing.

2A. Signing in Windows
Attached to the first post is a zip file named WinSign.zip . Download and extract to your Android-SDK/tools directory. To sign a ROM.zip file
I.Copy the ROM.zip file to the Android-SDK\tools\sign folder.
II.Open the Windows command prompt and navigate to your Android-SDK\tools\sign folder
III.At the prompt type:

java -jar signapk.jar testkey.x509.pem testkey.pk8 NAME-OF-ROM.zip NAME-OF-ROM-signed.zip

This will create a signed copy of your ROM, inject the rsa keys to allow flashing and place it in the folder alongside your original ROM.zip. The new file is ready to copy to the SD card for flashing and testing.

6. Conclusion
I hope you came away from this with a little more understanding. Of course there are many more tricks when it comes to ROM cooking. This should get you started, allow you to practice writing update-scripts, signing and flashing, as well as making your own theme ROMs to share with others. I'm always game to talk turkey with other budding Android developers, so head on over to the Hacking forums and lets get to cooking!

7. Disclaimer
Hacking by it's very nature can really bork things up. While all attempts were made to ensure this document is accurate, make sure you understand the following:
I, nor anyone associated with this website can be responsible for harm or damage resulting from the modification of your Android device. Any modifications to the hardware or software on your Android device is done at your own risk. Please be aware of the consequences of these actions and above all else, ask questions and find the answers you need before you attempt any of the ideas mentioned here.

This document is not protected under the GNU FDL and will be the property of Android Central and Smartphone Experts once published.
Portions of the attached binaries are provided by Lox_Dev@xda and myself. Many thanks go out to the Android development community and all it's members.