[SCRIPT] Migrate data to ext (0.4 - 4/23/11). An alternative to data2ext and Link2SD

[Delphis]

Not been happy with this, so I am reworking things

First off, make a nandroid backup

I have worked on this on my own phone but that's about it. I am using a Xionia kernel (010) (boot.img) on a Stock ROM. Customized with a Rodimus theme but otherwise a stock LG Optimus V. Your kernel needs to provide /etc/init.d support and also support for the filesystem you're wanting to use, e.g. ext3. The Xionia kernel with which I have tested this, provides those.

Attached is a flashable .zip that will install a script to do the following:

- mount the ext partition (as ext3 .. so edit the file if you need something else)
- move the directories app, data, local, misc and system from the /data partition to those named directories on the ext partition
- migrate and link dalvik cache entries for the /data/app programs to the ext partition. (IMPORTANT: You need to erase Dalvik cache after install)
- all directories linked are moved to a 'bak' directory so you can undo it. (See below for how to reclaim space after installation)
- any application and its data are then on your ext partition on the sd card, so no more storage space worries and nothing for you to do to maintain it!

CAVEATS:
- If there's no sd card, the phone will likely not boot. I hope to change this in an upcoming release.

TODO:
- get an e2fsck binary so the filesystem can be checked and fixed before mounting
- contingency plan for no sd card and/or failed mount so the phone boots, albeit with a 'blank slate', but it's better than not booting at all.
- separate preferences file, so customizations can be kept across updates

KNOWN BUGS:
- None?
- This works for me, that's all my testing I can do... if you find a bug, tell me

INSTALLATION

Boot to recovery (home, vol down, power) and make a nandroid backup

If you do not have an ext partition already:
Reboot the phone again, and back up the contents of your SD card (the FAT32 part) which will preserve your nandroid backups. Do this by connecting via USB and copying contents to your computer.
Boot to recovery again and partition/format the SD card to provide an ext3 partition - That will erase ALL your SD card contents!
You may wish to boot the phone to make sure it still comes up right. Recommended that you restore your SD card contents to the FAT32 part now.

In recovery, install the flashable zip file. This will also remove the Link2SD mount script automatically if it exists.

Your phone should boot and things are linked now. Wait, you're not done yet!

Restore your SD card contents if you have not done so already. That will keep your nandroid backups available.

Remove the Link2SD application if you had that installed before you do anything further, else you risk losing your applications

Shut down the phone, boot to recovery and make another nandroid backup.

In recovery, go to advanced menu and 'Wipe Dalvik Cache'. This is important that you do this before booting the phone again or it will likely not come up.

Once done, reboot the phone. It should start up correctly.

Now you're done and can install any applications with no decrease of storage size.

To reclaim storage, you will need to erase the '/data/bak' folder, either via Terminal Emulator or by Root Explorer.

Steps in Terminal Emulator:
$ su
# cd /data
# rm -r bak

Now you're done. Enjoy being able to install anything you like without worrying about storage space!

STORAGE SPACE

Please be aware that since we are LINKING from your original /data partition to directories on the ext partition you will not see the storage space of the ext partition when yo go to 'SD Card Storage' in the main settings menu. That's because the phone does not know about this extra storage space we've made available. Your original /data partition (that's part of the phone) is still there and mounted, we are just redirecting reads and writes to the SD card's ext partition. If you are really curious, go to Terminal Emulator and type 'df'. You will see your ext partition as '/data/sdext2' and can see how much is in use.

WHY?

I hope it helps someone manage their storage space better. For myself I wrote this because I was tired of using (albeit the very easy-to-use) Link2SD to link every program I installed. I was also thinking how every time I install an app to try it out it wears on the internal memory of the phone (that I cannot replace) where I would rather have it use the sd card (which can be replaced cheaply). I could not get the data2ext script working, so I wrote this as a way to get the best of both worlds. The /data partition is mounted as part of the init.rc script which is contained in the ramdisk inside boot.img, so it is very hard to edit to change it to a different location. That, I think, is why the original data2ext did not work for me. This method does, so I thought I'd share it.

The script can be invoked as 'migrate_dalvik_cache' provided you have root access and Terminal Emulator or SSH access to your phone. It will provide the same clean-up operations for the dalvik-cache without needing to reboot your phone. For the most part, it's unlikely you'll need to run it. The command name is important, don't re-run the script in /etc/init.d once the phone is booted.

So, please be careful with it, I hope it works Please report successes or failures.

VERSION HISTORY
0.1 - Initial release
0.2 - Mount options for better usage on flash storage, helps with responsiveness too
0.3 - Moved to ext3, bugfixes for setup of command line invocation
0.4 - Bugfixes for moving app directory

 

[mmarz]

Great work! Thanks for this! One suggestion for a future release:

1. Keep a copy of core programs on the phone
2. When booting and the script detects the ext partition is missing or corrupt, it defaults back to the core programs location.

This will ensure that when switching micro-sd cards or having micro-sd card corruption, you won't be left without a functioning phone.

Keep up the good work!

 

[Delphis]

Great work! Thanks for this! One suggestion for a future release:

1. Keep a copy of core programs on the phone
2. When booting and the script detects the ext partition is missing or corrupt, it defaults back to the core programs location.

This will ensure that when switching micro-sd cards or having micro-sd card corruption, you won't be left without a functioning phone.

Keep up the good work!

Thanks for the feedback and encouragement

Yes, I was wanting things to 'fail safe'. I was pondering this the other day the best way to do it. The init.rc scripts set up a default environment if the directories don't exist. I would need to remove the symlinks and recreate that default environment. With checking to see if we have the different dirs (app,data etc.) on our ext partition before copying it does attempt to not overwrite things if a default setup was on the phone. One big reason for moving the system directory is so that the package list does not get hosed should the ext partition go away. I may have to tinker with that and see how to make it work nicely.

By the way, since you make update zips all the time.. how do I go about signing them?

 

[mmarz]

Here you go: Auto-sign.rar

Rename the zip to update.zip and place it in the same folder. Run the bat file.

Also, since there are 1000 different app to ext scripts out there, I would personally make this Optimus V specific. That might eliminate the headache of making it super robust. We only have a handful of roms to work with, as long as it works on those, I'm happy.

 

[Delphis]

Here you go: Auto-sign.rar

Rename the zip to update.zip and place it in the same folder. Run the bat file.

Also, since there are 1000 different app to ext scripts out there, I would personally make this Optimus V specific. That might eliminate the headache of making it super robust. We only have a handful of roms to work with, as long as it works on those, I'm happy.

Thanks for that signing thing, that's cool. Since I am a Linux user (Ubuntu on my laptop ) I wrote an equivalent shell script that can be called with a zip file name as an argument and then it works out the proper output filename. I've attached the re-zipped bundle here (your thing plus my Linux bash script) in case anyone else wants it.

Yea, I know the script for moving things to ext isn't a new idea, I just never found something I liked and that worked like I wanted. I've re-uploaded a signed version with a change that I think helps with responsiveness and is better for flash media, the noatime option to mount. I think you posted a script of 'tweaks' that does that too for all partitions. I will need to look again to see if the RodimusROM I'm using already does that. Anyway, I digress... thread merging..oops!

 

[Takenover83]

Is this the only line that needs changed if I want to use ext4?

FSTYPE=ext4

 

[Delphis]

Is this the only line that needs changed if I want to use ext4?

Yes, that's it. I hope to have that available in future in another file that won't get overwritten with updates.

 

[Takenover83]

I keep getting this error when trying to install the zip in recovery. I have tried ext4 and ext2 partitions. Not sure what is on line 8, but that is whats stopping me. Attempting this on Bumblebee ROM 1.06. (Froyo 2.2.1)

edit: Maybe line "symlink /etc/init.d/10_migrate_data_to_ext"?

 

[Delphis]

I keep getting this error when trying to install the zip in recovery. I have tried ext4 and ext2 partitions. Not sure what is on line 8, but that is whats stopping me. Attempting this on Bumblebee ROM 1.06. (Froyo 2.2.1)

edit: Maybe line "symlink /etc/init.d/10_migrate_data_to_ext"?

that's the 'update-script' that's failing, the shell script isn't even running yet. I'll look into it. Thanks for giving it a go

 

[Delphis]

Please re-download it now and try it.. sorry about that. I must have not been paying attention when I did the update-script. I have not had luck sorting out the bug, although there were obvious problems with the first iteration of it. I've commented out the linking for command line use for now until I can get it to work. It was an afterthought and the update script is not doing what I want. So, you can install the currently available version and it should install fine for you

 

[Takenover83]

K thanks for the quick update. I will give it a shot. I will update this post if I get it to work.

Edit: Zip installed OK (even after I edited it to be ext4, just turned verification off). I have my ext4 partition setup that I created in recovery. I am still not showing the extra space when I look at Settings>SD Card & Phone Storage. Am I missing a step?

Edit2: Think I will hold off till a few more people test this.

 

[Delphis]

I am still not showing the extra space when I look at Settings>SD Card & Phone Storage. Am I missing a step?

No, but I don't believe you understand what the script is doing. I will update the main post after this.

Since we are symlinking from the /data partition the size reported will be that of the /data partition and not the sd card's ext partition.

Upon install, as mentioned already, the current app, data directories are moved to 'bak' under /data. If things look like they are working correctly then you can remove the 'bak' directory and recover the space since /data/app now is a link to the app directory on the sd card's ext partition. Installations will go direct to there now so you won't see you available space in /data decrease.

The phone's size app doesn't know about the sd's ext partition. You can see it though from a terminal by typing 'df'.

 

[Takenover83]

I may get around to tinkering with this some more today. You did clear up a few question's I had. Hopefully that e2fsck and libs we got working, will help with unmounting cleanly.

 

[Kreatur3]

Only reason I'm sticking with Link2SD is I really like being able to see where things are. I know it shouldn't really matter much but I'm a visual person. I did try to get something like this to work before trying Link which is how I ended up using it.

 

[Takenover83]

OK, still trying to get ext4 up and running. My partition size in 384MB, created with recovery-thunderc-v-1.2.0-GNM. Some of the other recoveries do not support creating ext3, or ext4 partitions. But this one does.

I edited the script. It installed just fine, once I turned verification off.

FSTYPE=ext4

This is the output of df.

C:\android\platform-tools>adb shell
# df
df
/dev: 217756K total, 0K used, 217756K available (block size 4096)
/cache: 112768K total, 3036K used, 109732K available (block size 4096)
/system: 168448K total, 99968K used, 68480K available (block size 4096)
/data: 183168K total, 120808K used, 62360K available (block size 4096)
/persist: 1536K total, 1156K used, 380K available (block size 4096)
/mnt/asec: 217756K total, 0K used, 217756K available (block size 4096)
/mnt/sdcard: 1542864K total, 434280K used, 1108584K available (block size 4096)
/mnt/secure/asec: 1542864K total, 434280K used, 1108584K available (block size 4
096)
/mnt/asec/com.rovio.angrybirdsrio-1: 14536K total, 12756K used, 1780K available
(block size 4096)
#

Is it working, lol? I don't think it is, as I removed my SD card and rebooted, and everything still looked the same. I think I would of got some errors or something with the SD card removed, or some apps missing, but nope. Everything was still there.

This is on Harmonia 1.0. Not sure it has init.d script support in the kernel.
Edit: I might be running into a different error here. I think the ext4 module needs loaded for the kernel I am using.

 

[Delphis]

Is it working, lol? I don't think it is, as I removed my SD card and rebooted, and everything still looked the same. I think I would of got some errors or something with the SD card removed, or some apps missing, but nope. Everything was still there.

This is on Harmonia 1.0. Not sure it has init.d script support in the kernel.
Edit: I might be running into a different error here. I think the ext4 module needs loaded for the kernel I am using.

Looks like it's doing nothing at all :> .. which would seem to indicate that it's either not running (/etc/init.d support) or cannot mount the filesystem, since if it detects nothing is mounted it pretty much bails out. Have you tried formatting the partition as ext3 and see if that makes a difference? For our purposes here there's not much difference.

Also, I have redone the main post to give clearer installation instructions. I have performed this on another phone now so I learnt a bit more about potential pitfalls and think I've documented it better.

 

[Delphis]

Only reason I'm sticking with Link2SD is I really like being able to see where things are. I know it shouldn't really matter much but I'm a visual person. I did try to get something like this to work before trying Link which is how I ended up using it.

That's fair enough and everyone is very welcome to their opinion, and I respect yours I'm just lazy I guess and got tired of tinkering with Link2SD, lol.

 

[Delphis]

I may get around to tinkering with this some more today. You did clear up a few question's I had. Hopefully that e2fsck and libs we got working, will help with unmounting cleanly.

The e2fsck will only help with checking the filesystem for errors. Unmounting cleanly is part of the shutdown scripts and I've not had a chance to look into where the appropriate place to invoke a 'umount' command would be. It needs to be unmounted (umount) when it's no longer in use. All Linux distributions unmount all mounted filesystems, regardless of when they were mounted. It seems the Android setup only unmounts specific ones and since it doesn't know about the ext partition, it doesn't unmount it properly. Journalled filesystems are able to cope with this sort of 'abuse', which is why I have hastily moved to ext3, whereas ext2 would have worked great if it was handled properly by the OS.

 

[Kreatur3]

That's fair enough and everyone is very welcome to their opinion, and I respect yours I'm just lazy I guess and got tired of tinkering with Link2SD, lol.

Don't you mean you're more driven, not lazy, since this takes more effort than Link? I'm the lazy one, you are no such thing. lol

Takenover83 just so you know the CWMA recovery version .6, when partitioning the sd card, creates it in ext3. Not sure if this helps you or not but thats the ext it creates. While I could partition my sd card with thunderc I couldn't ever get it to mount so I switched. Hope this helps or something.

 

[anon(235100)]

have you seen Ungaze's Interactive Data2sd post on XDA forum? A few days ago I was pm'ing back & forth with Ungaze about porting his tweak over to our phone. He might be a good resource.