B2 problems data recovery options

[damienjm]

Hi,

I'm at my wits end and my lack of knowledge about this particular area is hampering my efforts. I've left my B2 unattended (and I've therefore forgotten how I administered it) almost exclusively since I bought it many years ago. It turns out the backup I'd been doing to an external disk had stopped working some time ago. It was not until I discovered that B2 was having problems that I realised this, so I'm entirely dependent upon the B2 internal disk and the Bubba storage drive I bought for RAID purposes to attempt to recover all photos and videos of my two kids first 2 years. Stupid I know but I thought I had it all covered!

1TB B2 and 1TB Bubba eSATA storage configured with the default RAID1 option. Long story short: I cannot get access to it by the network or webadmin, or, one the very brief occasions when I do, it responds very slowly and eventually stops responding. . I finally resorted to creating a USB rescue disk. I mounted /dev/sda1 into /mnt as advised by one of the tutorials or readmes I found. I don't have enough subject matter knowledge to figure out how to mount the /home from the internal disk, to know if my data is gone or not.

In /mnt/varl/log/syslog there are errors that to my untrained eye look like errors with the attached external drive but none of the searches I did threw up any clear answers.

Code: Select all

Jan 21 20:46:10 bubba kernel: ata1: EH complete
Jan 21 20:46:27 bubba kernel: ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 act
ion 0x0
Jan 21 20:46:27 bubba kernel: ata1.00: BMDMA2 stat 0x80c0009
Jan 21 20:46:27 bubba kernel: ata1.00: failed command: READ DMA EXT
Jan 21 20:46:27 bubba kernel: ata1.00: cmd 25/00:08:91:53:3e/00:00:66:00:00/e0 t
ag 0 dma 4096 in
Jan 21 20:46:27 bubba kernel:          res 51/40:00:91:53:3e/00:00:66:00:00/e0 E
mask 0x9 (media error)
Jan 21 20:46:27 bubba kernel: ata1.00: status: { DRDY ERR }
Jan 21 20:46:27 bubba kernel: ata1.00: error: { UNC }
Jan 21 20:46:27 bubba kernel: ata1.00: configured for UDMA/100
Jan 21 20:46:27 bubba kernel: ata1: EH complete
Jan 21 20:46:27 bubba kernel: ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 act
ion 0x0
Jan 21 20:46:32 bubba kernel: ata1.00: BMDMA2 stat 0x80c0009
Jan 21 20:46:32 bubba kernel: ata1.00: failed command: READ DMA EXT
Jan 21 20:46:32 bubba kernel: ata1.00: cmd 25/00:08:91:53:3e/00:00:66:00:00/e0 t
#
Jan 21 20:46:32 bubba kernel:          res 51/40:00:91:53:3e/00:00:66:00:00/e0 E
mask 0x9 (media error)
Jan 21 20:46:32 bubba kernel: ata1.00: status: { DRDY ERR }
Jan 21 20:46:32 bubba kernel: ata1.00: error: { UNC }
Jan 21 20:46:32 bubba kernel: ata1.00: configured for UDMA/100
Jan 21 20:46:32 bubba kernel: ata1: EH complete
Jan 21 20:46:32 bubba kernel: ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 act
ion 0x0
Jan 21 20:46:32 bubba kernel: ata1.00: BMDMA2 stat 0x80c0009
Jan 21 20:46:32 bubba kernel: ata1.00: failed command: READ DMA EXT
Jan 21 20:46:32 bubba kernel: ata1.00: cmd 25/00:08:91:53:3e/00:00:66:00:00/e0 t
ag 0 dma 4096 in
Jan 21 20:46:32 bubba kernel:          res 51/40:00:91:53:3e/00:00:66:00:00/e0 E
mask 0x9 (media error)
Jan 21 20:46:32 bubba kernel: ata1.00: status: { DRDY ERR }
Jan 21 20:46:32 bubba kernel: ata1.00: error: { UNC }
Jan 21 20:46:32 bubba kernel: ata1.00: configured for UDMA/100
Jan 21 20:46:32 bubba kernel: sd 0:0:0:0: [sda] Unhandled sense code
Jan 21 20:46:32 bubba kernel: sd 0:0:0:0: [sda] Result: hostbyte=0x00 driverbyte
=0x08
Jan 21 20:46:32 bubba kernel: sd 0:0:0:0: [sda] Sense Key : 0x3 [current] [descr
iptor]
Jan 21 20:46:32 bubba kernel: Descriptor sense data with sense descriptors (in h
ex):
Jan 21 20:46:32 bubba kernel:         72 03 11 04 00 00 00 0c 00 0a 80 00 00 00
00 00
Jan 21 20:46:32 bubba kernel:         66 3e 53 91
Jan 21 20:46:32 bubba kernel: sd 0:0:0:0: [sda] ASC=0x11 ASCQ=0x4
Jan 21 20:46:32 bubba kernel: sd 0:0:0:0: [sda] CDB: cdb[0]=0x28: 28 00 66 3e 53
 91 00 00 08 00
Jan 21 20:46:32 bubba kernel: end_request: I/O error, dev sda, sector 1715360657

Jan 21 20:46:32 bubba kernel: ata1: EH complete
Jan 21 20:46:32 bubba kernel: EXT3-fs error (device md0): ext3_get_inode_loc: un
able to read inode block - inode=105988097, block=211976194
Jan 21 20:46:32 bubba kernel: ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 act
ion 0x0
Jan 21 20:46:32 bubba kernel: ata1.00: BMDMA2 stat 0x80c0009
Jan 21 20:46:32 bubba kernel: ata1.00: failed command: READ DMA EXT
Jan 21 20:46:32 bubba kernel: ata1.00: cmd 25/00:08:91:53:3e/00:00:66:00:00/e0 t
ag 0 dma 4096 in
Jan 21 20:46:32 bubba kernel:          res 51/40:00:91:53:3e/00:00:66:00:00/e0 E
mask 0x9 (media error)
Jan 21 20:46:32 bubba kernel: ata1.00: status: { DRDY ERR }
Jan 21 20:46:32 bubba kernel: ata1.00: error: { UNC }
Jan 21 20:46:32 bubba kernel: ata1.00: configured for UDMA/100
Jan 21 20:46:32 bubba kernel: ata1: EH complete

What I would like to be able to do is assess whether the internal or external drive might have errors on it so that I can focus on getting my data off the other. At the moment, I cannot locate my data on the mounted /dev/sda1 and I assume this is because the RAID setup means /home becomes dependent upon both being connected at the same time?? Is there any way I can tell if either of the drives are corrupted|ok and then take a look to see where my data might be.

By doing

Code: Select all

mount /dev/sda1 /mnt

Am I doing the right thing to gain access to the contents that would previously have been mounted as /home?

My original (/mnt/etc/fstab) is:

Code: Select all

/dev/md0        /home        auto        defaults        0        2
/dev/sda1        /        ext3        noatime,defaults        0        1
/dev/sda3        none        swap        sw        0        0
/proc        /proc        proc        defaults        0        0
UUID=1C0E-2567        /home/storage/extern/TOSHIBA_EXT-2        auto        defa
ults,umask=0002,uid=1000,gid=100        0        0
usbfs        /proc/bus/usb        usbfs        defaults        0        0

This is what suggests to me that my data can only be accessed with the external drive mounted also, but that's just a wild guess.

Yes, I know that some of what I'm putting forward probably looks stupid but I'm no expert and my unix knowledge is all but forgotten since I became a manager instead of someone who actually does work

Any help appreciated, since I don't know what information to provide.

[damienjm]

I hoped my question wouldn't be that challenging. Perhaps it's the detail I included in my post.

I know that it's probably frustrating to deal with someone whose knowledge is so limited but is there anyone that could suggest how I could go about investigating whether I have any possibility to recover my data?

I've been reading a thread about malicious access to some people's B2 and B3 systems. If I try to "remount" the eSata drive and start up the RAID configuration, is there any risk that even if the data is ok on it, that it could get deleted if there has been some malicious activity that caused my original problems?

[RandomUsername]

I haven't read your post fully (didn't see it when originally posted) and I have no experience of using the B2/3 in a RAID setup so I'm not sure how that changes things.

But, If you get a system running desktop Linux (Ubuntu live CD for example) and plug the hard drives into it with a USB/eSATA adapter, you can install some software called photorec that, in my experience, does an amazing job of recovering files from dodgy disks. This tutorial should help (although is quite old now so some steps and screenshots may differ slightly) - http://www.psychocats.net/ubuntucat/rec ... etedfiles/.

When I get some more time I will try to more thoroughly read your post. I hope you get them back, the photos of my kids growing up are probably the most valuable things I own.

And a lesson for everyone; it's no good having backups in place if you don't regularly test them.

[Ubi]

isnt the thing just sluggish because it is rebuilding the RAID?

Code: Select all

cat /proc/mdstat

PS: RAID is NOT a backup!!!!!

[RandomUsername]

What I would like to be able to do is assess whether the internal or external drive might have errors on it so that I can focus on getting my data off the other. At the moment, I cannot locate my data on the mounted /dev/sda1 and I assume this is because the RAID setup means /home becomes dependent upon both being connected at the same time??

The problem is that /home is on a logical volume and so cannot be mounted with a simple mount command.

(N.b. the following is from memory so some steps may not be exactly right or missing, report back if you can't work any part of it out):
If you boot from a rescue stick, you need to make the volume available first with the command

Code: Select all

lvchange -ay bubba

Then mount the volume:

Code: Select all

mount /dev/mapper/bubba-storage /mnt

[Edit] You may need to run

Code: Select all

pvscan

Code: Select all

vgscan

and/or

Code: Select all

lvscan

before the logical volume is detected.

[damienjm]

Thanks for all the suggestions. I'll spend some time this afternoon trying them out.

I do realise that RAID is not a backup, unfortunately I realised that my backup had gone south at the time I realised that B2 was having problems. Wondering whether RAID is even worth the hassle in future, when I could use that disk as part of my backup rotation...

[damienjm]

I'm back but without much progress. I hade to make up a new rescue usb because no prodding, poking or begging would convince B2 to boot up again with the one that worked previously!

I've been trying to understand a little more about how logical volumes work in order to diagnose why your suggestions didn't work for me. The lvscan, pvscan or vgscan didn't return any results, although they threw up read failed errors on /dev/sda3. I've include some relevant detail from some of the things I tried below in the event that they might help someone to diagnose or suggest next steps.

Sequence of events:
Boot from rescue USB with the e-SATA disk attached
I didn't attempt to mount anything, as I assumed from the previous response that this is futile
I then ran the following commands to get a picture of whether the disks are being recognised (or are completely hosed):

Code: Select all

# fdisk -l 

Disk /dev/sda: 1000.2 GB, 1000204886016 bytes 
255 heads, 63 sectors/track, 121601 cylinders 
Units = cylinders of 16065 * 512 = 8225280 bytes 
 
   Device Boot      Start         End      Blocks  Id System 
/dev/sda1               1        1217     9775521  83 Linux 
/dev/sda2            1218      121457   965827800  fd Linux raid autodetect 
/dev/sda3          121458      121601     1156680  82 Linux swap 

Disk /dev/sdb: 4102 MB, 4102028288 bytes 
255 heads, 63 sectors/track, 498 cylinders 
Units = cylinders of 16065 * 512 = 8225280 bytes 

   Device Boot      Start         End      Blocks  Id System 
/dev/sdb1               1         499     4005855+  c Win95 FAT32 (LBA) 
Partition 1 has different physical/logical endings: 
     phys=(497, 254, 63) logical=(498, 181, 1) 

Disk /dev/sdc: 1000.2 GB, 1000204886016 bytes 
255 heads, 63 sectors/track, 121601 cylinders 
Units = cylinders of 16065 * 512 = 8225280 bytes 

   Device Boot      Start         End      Blocks  Id System 
/dev/sdc1               1      121602   976762583+ ee EFI GPT 

I tried scanning for the logical, physical and volume groups:

Code: Select all

# pvscan -v
File descriptor 5 left open
File descriptor 7 left open
    Wiping cache of LVM-capable devices
    Wiping internal VG cache
    Walking through all physical volumes
  /dev/sda3: read failed after 0 of 4096 at 0: Input/output error
  /dev/sda3: read failed after 0 of 4096 at 4096: Input/output error
  /dev/sda3: read failed after 0 of 4096 at 0: Input/output error
  No matching physical volumes found
# lvscan -v
File descriptor 5 left open
File descriptor 7 left open
    Finding all logical volumes
  /dev/sda3: read failed after 0 of 4096 at 0: Input/output error
# vgscan -v
File descriptor 5 left open
File descriptor 7 left open
    Wiping cache of LVM-capable devices
    Wiping internal VG cache
  Reading all physical volumes.  This may take a while...
    Finding all volume groups
  /dev/sda3: read failed after 0 of 4096 at 0: Input/output error
  /dev/sda3: read failed after 0 of 4096 at 4096: Input/output error
  /dev/sda3: read failed after 0 of 4096 at 0: Input/output error

Out of curiosity I also did a lvmdiskscan:

Code: Select all

# lvmdiskscan 
File descriptor 5 left open 
File descriptor 7 left open 
  /dev/sda3: read failed after 0 of 4096 at 0: Input/output error 
  /dev/sda3: read failed after 0 of 4096 at 4096: Input/output error 
  /dev/ram0  [       32.00 MB] 
  /dev/ram1  [       32.00 MB] 
  /dev/sda1  [        9.32 GB] 
  /dev/ram2  [       32.00 MB] 
  /dev/ram3  [       32.00 MB] 
  /dev/sda3: read failed after 0 of 4096 at 0: Input/output error 
  /dev/sda3: read failed after 0 of 4096 at 0: Input/output error 
  /dev/ram4  [       32.00 MB] 
  /dev/ram5  [       32.00 MB] 
  /dev/ram6  [       32.00 MB] 
  /dev/ram7  [       32.00 MB] 
  /dev/ram8  [       32.00 MB] 
  /dev/ram9  [       32.00 MB] 
  /dev/ram10 [       32.00 MB] 
  /dev/ram11 [       32.00 MB] 
  /dev/ram12 [       32.00 MB] 
  /dev/ram13 [       32.00 MB] 
  /dev/ram14 [       32.00 MB] 
  /dev/ram15 [       32.00 MB] 
  /dev/sdb1  [        3.82 GB] 
  /dev/sdc1  [      931.51 GB] 
  0 disks 
  19 partitions 
  0 LVM physical volume whole disks 
  0 LVM physical volumes

However, since I'm booting up from a rescue disk, I'm wondering whether it will even be possible to find lv details if they are dependent upon what would be available either when the drives are mounted or linux is running from the disk in bubba, as opposed to the rescue disk. Perhaps I'm being naive but wont the details of volume groups only be available in the configuration files on disk - or are they written into the MBR or disk partition details?

As you can see, I'm completely clueless when it comes to this and therfore possibly overthinking it. I can't find any help on that topic because all lvm administration help assumes working on a running linux instance on an already mounted disk. Is that a red herring, or do I actually need to take some additional steps first?

UPDATE*
Ok, so I started to think through what I wrote and this led me to another avenue of investigation. I found an article elsewhere (http://www.howtoforge.com/recover_data_ ... partitions) that suggested that I needed to use mdadm to start the RAID setup. This is a challenge because it's not on the rescue USB. Would this be a pre-requisite to the steps already suggested?

That leaves me with 2 questions about my recommended next steps:
1. Would I be better to focus on the /etc/sda3 read errors first and might that allow me continue with the suggestions made by RandomUserName (ignoring mdmadm)OR;
2. Would I be better to just download a Debian LiveCD image, pull the disks from B2 and attempt to more closely follow the suggestions in http://www.howtoforge.com/recover_data_ ... partitions?

[Ubi]

Forget about mdadm, it is only required for raid0 or raid5
The raid1 you have creates two perfectly readable partitions than can also be mounted standalone. Doing this will break the consistency of the raid, but that is not the biggest issue you have it seems.

Als the content of the lvm is witten into the lvm, there is no config file. However there was something about limited lvm support in the rescue disk. There has been trouble with this combination before. Could be solved i dont know

[damienjm]

Ok, great that I know I can ignore mdadm.

So what am I doing wrong, or better still, what do I need to do to be able to mount and read the data on the disks?

What I've tried doesn't work and I don't know whether what I've done is incorrect or what I've tried is failing because of the read failures on sda3. If,as you say lvm on the rescue disk is insufficient, would the recommendation be to take the disks out and attempt to mount them on another full running debian instance, for example?

[RandomUsername]

I don't remember any problems reading the lvm on the rescue stick, but it might be worth trying a live cd. It's worth investing in a SATA to USB cable.

[Gordon]

The noticeable thing here is that the lvm tools apparently do not scan sdc at all.

If you have a PC with an eSATA port you could try to connect the S1 to that machine. Don't worry if that machine does not run Linux, because you can use a live CD from e.g. Ubuntu. As an alternative, since the internal disk is apparently broken, why not exchange it with the disk from the S1? If the tools are limited to scanning sda only that should fix the issue, provided that the disk in the S1 is not also corrupted.


Just a thought:
It is possible to set filters on the lvm tools in /etc/lvm/lvm.conf. The common reason to set these is to exclude the cdrom, but what if the rescue system has that limited tot sda only?