How to recover data from "adb backup --twrp" archive? Been through hell and back
I have two backups I made using "adb backup --twrp" while my OnePlus one phone was in TWRP recovery 3.2.1-0 running LineageOS with a recent build (don't know exactly which one, but it was from Feb/March 2018). Each of the backups are about 6.0GB so they appear to be complete backups. I am trying to recover one app and its data (Signal/Textsecure) but the rest would be nice too. The backup files have .ab extensions. I am fairly android savvy and have been working on this for days, your help is VERY MUCH APPRECIATED! I have access to Windows and Linux.
I have tried the following:
-"adb restore filename.ab" while in TWRP recovery. It appears to restore (progress bar goes, it reports success, etc), but no apps are restored. I have tried this both with totally wiped partitions/system/data and with a fresh install of LineageOS. As far as I can tell, nothing happens. When doing a restore after a full wipe and trying to restart, TWRP warns that no OS is installed.
-Restoring using the restore menu in TWRP. Immediately upon clicking on the backup, the phone restarts and TWRP opens again. Not sure what's going on there.
-Opening the backups in Titanium Backup, Nandroid manager, and a few other apps. None of them worked. Titanium backup crashes instantly in recover from ADB, doesn't recognize the backups in nandroid recovery mode.
-Opening the .ab files in an archive manager, all of which report it isn't a valid tar archive. One time I was able to get a list of directories, and things looked pretty right, so I know at least some of the data should be there. Running variants of this command with different skip values, none of which produces a valid tar file dd if=file.ab bs=512 skip=1 of=test.tar
Questions:
- Is there a way to extract data from these backups in Linux or Windows outside of Android? From what I read, TWRP adds some stuff in the headers which makes them invalid tar files, and that doing "adb backup --twrp" is different from a regular TWRP backup.
- Why is TWRP crashing? Is there a log I can look at for this and how? Maybe i can find a way to diagnose it that way.
- Where does Signal store message data? I have a flat-file backup that may have this information depending on where it's stored.
- Can I somehow convert these backups to a different format that might be better-tolerated by TWRP or other apps?
- Any good utilities for reading a corrupted tar file? I think all this utility would need to do is skip through the junk at the start.
Related
I wanna start by saying that I have been searching and I can't find a similar post...forgive me if I was simply unsuccessful :/
So here's what happened.
Me - "Hey! I've got a rooted phone, but no custom ROM? I think I'll download one!"
I performed a Nandroid backup.
Flashed Macnut 14.
"Awesome, it works. Now I'm gonna restore my data."
I tried doing an advanced Nandroid restore, but it hung at "dalvik-cache".
I then tried doing a full Nandroid retore, but it still hung at "dalvik-cahe".
I then flashed Stock JK6 and repeated steps 5 and 6 with the same results.
I then downloaded and compiled unyaffs in order to get files out of the Nandroid .img files.
I tried "adb install" for apps and "adb push" for both data and apps. I've been getting error messages for both approaches (not syntax errors, but things like "permission denied," "not enough space," etc.), both while the phone is booted and when it's in recovery.
I've also tried moving the files to my SD card and then using "adb shell" to move them to their appropriate folders, but that didn't work either.
Besides these errors, I also had a very hard time figuring out where to put certain things.
So that's what happened...you guys are my final hope. I'm trying to figure out a foolproof way to restore (in order of priority):
Application data (settings, high scores in games, etc.).
Applications themselves (.apk files, preferably not overwriting newer versions of things like GMail, Youtube, etc. included in Stock JK6).
Text messages.
Phone settings.
I've learned my lesson...next time, I should use Titanium Backup.
Thanks,
Gerard
Personally I would:
1. ODIN to JK6
2. Root and use the modified 3e recovery
3. Install ROM Manager from the market and flash CWM Recovery
4. Then attempt to Nandroid restore
If you're not comfortable with the above Step 2 you can:
1. ODIN to JFD and root
2. Install ROM Manager from the market and flash CWM Recovery
3. CWM Flash JK6
4. Attempt to Nandroid restore
You went from eclair to froyo and when you restored your backup, CWR does not restore the kernel.
Try flashing the stock 2.1 kernel and then restoring your backup.
And yes, titanium backup & SMS Backup+ are your friends.
hmm... re-read... it's odd that a restore started and didnt finish. Most people get hung at the vibrant screen when they do not restore the kernel.
Sounds like you nandroid is corrupt, and if that is the case I'm afraid that I do not know the answer you seek.
Yeah, I don't think that a Nandroid backup is gonna do the job...I just tried again after ODIN.
Any other ideas?
EDIT: I was finally able to push all my applications back...However, I'm having trouble pushing the contents of "data/data/" and "datadata/". I keep getting errors like:
failed to copy 'data/com.android.browser/app_appcache/ApplicationCache.db' to 'datadata/com.android.browser/app_appcache/ApplicationCache.db': No such file or directory
Click to expand...
Click to collapse
How can I evade this error? :/
You have to flash a kernel as well cause nandroid won't replace that
Sent from my GT-I9000 using XDA App
Are you trying to adb from linux or windows? If you are on linux try this:
1) sudo adb kill-server
2) sudo adb devices
3) now push your apps
Also I found out the hard way, do not restore apps or system data from nandroid backup, I always had a corrupted system with a lot of force closes. Always use Titanium Backup and SMS Backup.
Hope this helps
brnbock said:
Are you trying to adb from linux or windows? If you are on linux try this:
1) sudo adb kill-server
2) sudo adb devices
3) now push your apps
Also I found out the hard way, do not restore apps or system data from nandroid backup, I always had a corrupted system with a lot of force closes. Always use Titanium Backup and SMS Backup.
Hope this helps
Click to expand...
Click to collapse
I'm actually on a Mac. I tried your method but I still got the same results...is Mac close enough to Linux (since it's Unix), or do you think I should attempt it on Linux as well?
DEFINITIONOFREAL said:
You have to flash a kernel as well cause nandroid won't replace that
Sent from my GT-I9000 using XDA App
Click to expand...
Click to collapse
I don't mean to sound like a douche...but isn't that what s15274n was saying?
ya try it on linux, Mac is BSD based but I'm not sure how permissions are set. BSD is close to linux but there are some major diff and some commands are diff.
So far, for me Nandroid on the Vibrant has been completely useless. I don't even bother any more.
I use Titanium to back up my apps and my SMS messages, and it only takes a couple minutes to re-do the settings once I install a new ROM.
This is way different than my experience with the G1, where I could easily restore a Nandroid backup if I didn't like a new ROM install. On the Vibrant it's easier to just Odin back to a stock ROM and redo all my settings using TB.
brnbock said:
ya try it on linux, Mac is BSD based but I'm not sure how permissions are set. BSD is close to linux but there are some major diff and some commands are diff.
Click to expand...
Click to collapse
Yeah...tried this too after downloading and setting up everything on my Linux box (java, Android SDK, etc.). Same error as I got on my Mac
I think I'll take like one more suggestion from the community before I finally give up and start my phone from scratch...It's been like 3 days since I lost all my data, ugh.
Hello, while I was using my phone it rebooted and aparrently someting from my /data got corupted disabling my Rom from booting ever again.
So I created a nandroid via CWM6, installed a new rom and I'm now scratching my head on how to extract certain apps from that backup. Using CWM5 I could do this via the use of Titanium Backup, but now it's more complex than that CWM6 nandroids are unreadable from anything that exists out there. The only app which could analyze the backup correctly was the latest beta of "app extractor" but even that could not extract the apks *with* their data.
Basically my apps' data is precious and I wish to restore it at all costs, I have a nandroid of them, but the format is so unspeakably complex that it is basically useless. So the other "solution" was restore the nandroid and pull the whole /data partition in my PC via ADB, but I'm not sure what I can do with it.
So as it stands I now have two copies of my apps, one in a nandroid format another as it was pulled from the phone but I'm not sure how I can go on and restore them in a new rom. As I have found out CWM6 was quite a downgrade for/to those who backup/recover frequently, it's sad that I have found about it in such a disastrous way
i switched to twrp recovery (v2.6.3.0) recently. i usually do backups while the phone is still switched 'on' - i.e., without booting into recovery - by either of the following methods:
----------(i) using online nandroid backup app (v4.2.1)
----------(ii) executing the appropriate command after installing onandroid script (v9.20)
these backups are created in /sdcard/TWRP/BACKUPS/0000000000000000/. but, backups done from within recovery are created in /sdcard/TWRP/BACKUPS/Micromax_A116/.
'restore' from within recovery detects backups located in /sdcard/TWRP/BACKUPS/Micromax_A116/, but not /sdcard/TWRP/BACKUPS/0000000000000000/.
since i wanna be able to restore my backups with ease, rather than having to first move the backup from the former location to the latter location, i'd like help regarding the following:
----------=> saving the backups created by "online nandroid backup app" or "onandroid script" directly in /sdcard/TWRP/BACKUPS/Micromax_A116/
----------=> getting twrp recovery to detect backups in /sdcard/TWRP/BACKUPS/0000000000000000/
i'm a bit . somebody please help. thanks.
My phone ran into problems after I upgraded Magisk (I was on Canary build). No problem I figured, I will just restore from my TWRP backup.
... only issue was, I have never restored from a TWRP backup. And it turned out to be not as straightforward as I thought. I ended up learning a lot about the process to be able to successfully backup AND RESTORE. So I thought I might write something here to make it easier for you to do this, so you don't have to learn it the hard way like I did.
First I think most of us know data storage is not backed up by default. The partitions selected by TWRP for you (i.e. system, data, vendor, and boot) are sufficient to save an image of your OS plus the installed apps, but not app data (specifically /data/media and everything underneath, so actually this is dependent on how each app stores its data, some or all of an app's data may be stored in /data/data and not /data/media. /data/data IS backed up by TWRP.), and the excluded portion contains your own personal data as well, i.e. the internal storage.
Simply put, if you want a so-called 'full backup', or a complete snapshot of your phone, you will want to do something more than simply a TWRP backup. See below.
Another thing to note is that the location you are backing up to (and I recommend external storage like a USB-OTG device) may *not* have enough space. And while backing up you may not even realize that you have run out of space (as there is no warning particularly with the tar command which I will detail below) therefore your backup may become corrupt ! Take extra precaution to validate the integrity of your backup, and make sure there is always enough space to create the backup. TWRP *does* warn you about lack of space though.
Last but not least, if you have a password (either pattern or numeric) to lock your device, after restoring it is gone. So first thing after restore is to go into settings to setup the password once again. Interestingly the fingerprint data is still there. Once you set up a password the fingerprints will reattach to it. For the apps that have fingerprint login registered, you will need to re-register as the app thinks you have updated your password/fingerprint.
If you don't set up your password after restoring, in TWRP it will just go straight into the main menu as if no password has been set, so basically anyone who knows how TWRP works will be able to access your data if they get a hold of your phone.
Oh one more thing. The format of the backup media (e.g. usb-otg) has to be ex-FAT. Very important. I got badly burned because I had FAT32. And like I said the backup process did NOT warn me at all that this could be a potential issue. Not the tar command anyway.
OK here is the HOW on BACKing UP:
Backup using TWRP should be easy enough. It is documented just above everywhere (restoring is another story however). Just make sure you mount your USB-OTG before starting your backup, and pick USB-OTG as your destination. Let TWRP choose the partitions to be backed up.
This will create a TWRP folder with subfolders, one of which containing the actual backup files.
On to your personal data - in TWRP (while your USB is still mounted), go to Advanced, Terminal, then issue this command: "tar -cpvzf /usb_otg/data_media_backup.tar.gz /data/media"
This will create a .tar.gz file. This is your own data (and like I said some apps write to this space too so it is also app backup).
Validate the integrity of the .tar.gz file with this command: "tar -tf /usb_otg/data_media_backup.tar.gz". If you see no error message then you are good.
Unmount your usb finger, and take it out before leaving TWRP. Otherwise the usb finger may need repairing when you access it in say Windows. And that's always a scary thing.
If you need to archive the backup somewhere in your PC etc, copy 2 things from your USB: The TWRP folder, or at least the subfolder that contains your TWRP backup files (copy the entire folder, or better yet the entire folder structure); and the .tar.gz file. This forms your complete backup image.
Here is the HOW on RESTORING:
Flash MIUI (or whatever your OS is), with matching version as the one in your backup image. i.e. you have a vanilla enviroment that is the same version as the one in your backup.
If for some reason you can't boot into your system afterwards, go into recovery (can be built in recovery, not necessarily TWRP), then wipe your DATA partition clean. Now you should be able to boot.
Boot up the vanilla system. When the first spalsh screen is shown ("welcome to the OS, let's start setting things up", or something like that) just long press the power button to shutdown or restart your phone. Then go into bootloader (lower volume + power buttons for my phone).
Install TWRP, reboot to recovery / TWRP, then do your TWRP restore. Don't forget to mount your otg-usb first ! if the folder structure is there in your usb, or even if you just have that one folder with the backup files underneath, the restore function will show that folder for you to pick. Just pick it then proceed with the restore.
in TWRP (while your USB is still mounted), go to Advanced, Terminal, then issue this command: "rm -r /data/media/0/*" (this is why step 3 was necessary)
if there are other folders underneath /data/media, repeat the rm command once for each of these folders
"cd /"
"tar -xvzf /usb_otg/data_media_backup.tar.gz" <-- this will restore your data !!
Reboot. Voala you have your backed-up system in action once again. Nothing in your phone is different than before the backup, except for the pattern password which is now gone. And you better set it up again right away.
Hope this helps !!
Hi! So, I have a newer Android 10 device with no TWRP support. I think my question is not model specific, so I'm posting generally, but if it matters, it's a OnePlus 7T Pro, bootloader unlocked of course.
Anyway, since there's no good custom recoveries for Android Q (none that I know of that can decrypt /data), I was trying to be safe and using a hacked (adb enabled) stock recovery to make backups from the command line. I have a nice tarball created in recovery mode with "tar cvf - /data/". It looks like it was made properly, I can unpack the files on back onto the device in recovery mode, and it's not corrupt or anything.
The problem is that naively trying to restore it back to my /data partition (e.g. tar xvf) after a wipe causes two problems:
1) after rebooting to recovery, the /data partition just looks encrypted again. If it was working properly, stock recovery can decrypt /data
2) normal system boot just bootloops.
If anyone has tried manually saving and restoring /data from a tarball, I'd appreciate any tips on getting it to work.
My best guess is that the ext4 encryption keys are the culprit - since after a factory reset the device creates new encryption keys, I think I need to figure out how to restore 99% of my backup without overwriting the fresh encryption keys that were created after wiping. I think I need to combine the new encryption keys with the other files in my tarball backup. If I can get past that, I might have to worry about things like SELinux xattrs as well, but one thing at a time.
If any experienced Android hackers know how filesystem encryption works in Android 10, I would appreciate a map of which files should not be overwritten. For example, I'm thinking I should probably avoid restoring some of these files:
Code:
/data/unencrypted/key
/data/unencrypted/key/encrypted_key
/data/unencrypted/key/keymaster_key_blob
/data/misc/vold/user_keys
/data/misc/vold/user_keys/de/0/keymaster_key_blob
/data/misc/vold/user_keys/de/0/encrypted_key
/data/misc/vold/user_keys/ce/0/current/keymaster_key_blob
/data/misc/vold/user_keys/ce/0/current/encrypted_key
/data/misc/systemkeys
/data/misc/keystore
/data/misc/keychain
/data/system/recoverablekeystore.db
[edit] spoke too soon
I ended up doing something janky and wrote a little shell script to repack the files from /data/data and /data/app into files that Titanium Backup could understand. I also figured out how to restore my SMS messages from /data/user_de. That was enough to recover the things I cared most about.
Still begs the question of what's the best method of doing backup and restores of /data on Android 10 when you don't have custom recovery but you do have Magisk, root, and stock recovery with adb. I want to keep hacking and playing with this phone so gotta figure something out.
I like Titanium Backup, have used it for many years, but the maintainance hasn't really kept up with Android development, theres a lot of gotchas like it hangs trying to restore multi APK packages. I still want to figure out something more seamless like a nandroid backup solution even if its from the command line. At least until TWRP eventually supports Android 10.
try backing up data partition in a external storage live usb or sd card
DrWowe said:
I ended up doing something janky and wrote a little shell script to repack the files from /data/data and /data/app into files that Titanium Backup could understand. I also figured out how to restore my SMS messages from /data/user_de. That was enough to recover the things I cared most about.
Still begs the question of what's the best method of doing backup and restores of /data on Android 10 when you don't have custom recovery but you do have Magisk, root, and stock recovery with adb. I want to keep hacking and playing with this phone so gotta figure something out.
I like Titanium Backup, have used it for many years, but the maintainance hasn't really kept up with Android development, theres a lot of gotchas like it hangs trying to restore multi APK packages. I still want to figure out something more seamless like a nandroid backup solution even if its from the command line. At least until TWRP eventually supports Android 10.
Click to expand...
Click to collapse
try backing up data partition in a external storage live usb or sd card, first you for encryption, then restore from external source after formatting phone storage