Post by Edmon E. Gabonilas on Jun 23, 2015 18:30:41 GMT 8
By xpirt, XDA Recognized Contributor
Introduction
You probably know already that starting from Android 5.x (Lollipop) compiled roms (aosp,cm,stock) are not compressed anymore the way they used to be on previous android versions. On previous versions all content inside /system folder that has to be extracted within our device was either uncompressed (simple /system folder inside our flashable zip) or compressed in a system.img file, which it is a ext4 compressed file; both of these, anyway, were readable and we could see all system files (app,framework, etc).
The problem comes in 5.x versions, this method is not used anymore. Why? Because roms started to be always larger, so it is necessary to compress them even more.
What does Android 5.x zips (full roms, but also otas) contain?
Android 5.x flashable zips are made this way:
boot.img (we all know what it is)
file_contexts
META-INF (folder containing scripts)
system.new.dat (compressed /system partition)
system.patch.dat
system.transfer.list (see explanation below)
What does updater-script contains then?
The updater-script uses a brand new function: block_image_update(), this method basically decompresses necessary files inside the device. Let's study it.
From google git source code, if we go inside the new file /bootable/recovery/updater/blockimg.c, we find at the end of it the registration of the function block_image_update() as the method BlockImageUpdateFn() which starts at line 254. Here finally we find all information we need to know about the decompression of the .dat file(s). First file we analyze is system.transfer.list which Google tells us:
Quote:
The transfer list is a text file containing commands to transfer data from one place to another on the target partition.
But what each line means?:
Quote:
First line is the version number of the transfer list; 1 for android 5.0.x, 2 for android 5.1.x
Second line is the total number of blocks being written
Third line is how many stash entries are needed simultaneously (only on version >= 2)
Fourth line is the maximum number of blocks that will be stashed simultaneously (only on version >= 2)
Fifth line and subsequent lines are all individual transfer commands.
Ok, but how to decompress the system.new.dat file?
howellzhu and luxi78, two guys from chinese development forums managed to write some binaries to decompress/compress this file. Big thanks to them for initial sources which I've compiled from!
You can use/modify these files and/or include them in your work as long as proper credits and a link to this thread are given.
If you have questions or problems write here
Are you building from source and want to revert to the old flashing method without DAT files?
Do this (thx @elluel):
1 - Delete this line
2 - Make sure you have this commit
Credits
howellzhu & luxi78 - for initial source codes
Windows 32/64 bit - Guide
Step 1 - Decompressing = DAT (sparse data) -> EXT4 (raw image)
We're now using sdat2img.exe binary, the usage is very simple:
sdat2img <transfer_list> <system_new_file> <system_ext4>
- <transfer_list> = input, system.transfer.list from rom zip
- <system_new_file> = input, system.new.dat from rom zip
- <system_ext4> = output ext4 raw image file
and a quick example:
sdat2img system.transfer.list system.new.dat system.img
by running this command you will get as output the file my_new_system.img which is the raw ext4 image.
Step 2 - Decompress EXT4 (raw image) -> OUTPUT folder -> Compress EXT4 (raw image)
Now you should either move to Ubuntu x64/x86 or follow the 2nd step from the Ubuntu guide on cygwin 32 bit.
Step 3 - Converting = EXT4 (raw image) -> DAT (sparse data)
Now we need the rimg2sdat.exe binary, the usage is very simple:
rimg2sdat <system_img>
- <system_img> = name of input ext4 raw image file
and a quick example:
rimg2sdat my_new_system.img
As you can see the output is composed by system.transfer.list, (system.patch.dat) & system.new.dat, ready to be replaced inside your Rom zip.
DOWNLOADs
sdat2img.exe
- mega.co.nz
make_ext4fs
- mega.co.nz
rimg2sdat.exe
- mega.co.nz
Introduction
You probably know already that starting from Android 5.x (Lollipop) compiled roms (aosp,cm,stock) are not compressed anymore the way they used to be on previous android versions. On previous versions all content inside /system folder that has to be extracted within our device was either uncompressed (simple /system folder inside our flashable zip) or compressed in a system.img file, which it is a ext4 compressed file; both of these, anyway, were readable and we could see all system files (app,framework, etc).
The problem comes in 5.x versions, this method is not used anymore. Why? Because roms started to be always larger, so it is necessary to compress them even more.
What does Android 5.x zips (full roms, but also otas) contain?
Android 5.x flashable zips are made this way:
boot.img (we all know what it is)
file_contexts
META-INF (folder containing scripts)
system.new.dat (compressed /system partition)
system.patch.dat
system.transfer.list (see explanation below)
What does updater-script contains then?
The updater-script uses a brand new function: block_image_update(), this method basically decompresses necessary files inside the device. Let's study it.
From google git source code, if we go inside the new file /bootable/recovery/updater/blockimg.c, we find at the end of it the registration of the function block_image_update() as the method BlockImageUpdateFn() which starts at line 254. Here finally we find all information we need to know about the decompression of the .dat file(s). First file we analyze is system.transfer.list which Google tells us:
Quote:
The transfer list is a text file containing commands to transfer data from one place to another on the target partition.
But what each line means?:
Quote:
First line is the version number of the transfer list; 1 for android 5.0.x, 2 for android 5.1.x
Second line is the total number of blocks being written
Third line is how many stash entries are needed simultaneously (only on version >= 2)
Fourth line is the maximum number of blocks that will be stashed simultaneously (only on version >= 2)
Fifth line and subsequent lines are all individual transfer commands.
Ok, but how to decompress the system.new.dat file?
howellzhu and luxi78, two guys from chinese development forums managed to write some binaries to decompress/compress this file. Big thanks to them for initial sources which I've compiled from!
You can use/modify these files and/or include them in your work as long as proper credits and a link to this thread are given.
If you have questions or problems write here
Are you building from source and want to revert to the old flashing method without DAT files?
Do this (thx @elluel):
1 - Delete this line
2 - Make sure you have this commit
Credits
howellzhu & luxi78 - for initial source codes
Windows 32/64 bit - Guide
Step 1 - Decompressing = DAT (sparse data) -> EXT4 (raw image)
We're now using sdat2img.exe binary, the usage is very simple:
sdat2img <transfer_list> <system_new_file> <system_ext4>
- <transfer_list> = input, system.transfer.list from rom zip
- <system_new_file> = input, system.new.dat from rom zip
- <system_ext4> = output ext4 raw image file
and a quick example:
sdat2img system.transfer.list system.new.dat system.img
by running this command you will get as output the file my_new_system.img which is the raw ext4 image.
Step 2 - Decompress EXT4 (raw image) -> OUTPUT folder -> Compress EXT4 (raw image)
Now you should either move to Ubuntu x64/x86 or follow the 2nd step from the Ubuntu guide on cygwin 32 bit.
Step 3 - Converting = EXT4 (raw image) -> DAT (sparse data)
Now we need the rimg2sdat.exe binary, the usage is very simple:
rimg2sdat <system_img>
- <system_img> = name of input ext4 raw image file
and a quick example:
rimg2sdat my_new_system.img
As you can see the output is composed by system.transfer.list, (system.patch.dat) & system.new.dat, ready to be replaced inside your Rom zip.
DOWNLOADs
sdat2img.exe
- mega.co.nz
make_ext4fs
- mega.co.nz
rimg2sdat.exe
- mega.co.nz