["SOLVED"] B3 inaccessible with limited functionality

[rojter]

Hi!

My B3 ignores me.

When doing my usual backup to my B3 this Sunday I noticed the file transfer after a couple of minutes just froze and would not continue. I restarted Explorer in Windows and also went to my B3 web interface and performed a reboot. After that reboot I can't reach the B3 web interface, can't reach it through SSH, can't reach my home shares from Windows and also the WiFi went down. However the DHCP still seems to be working since my laptop and desktop get their LAN IPs and I reach the internet. I can also ping it.

I'm not really sure where to go from here, so any ideas to the next step troubleshooting this are most welcome.

[johannes]

Sorry but this sounds as a partial disk failure. Does the led become blue when booted? If you leave it on (a few hours at least) does the HDD activity stop? If yes, can you perhaps reach it then?

If not I suggest you go on with rescue stick trying to figure out what has happened.

http://wiki.mybubba.org/wiki/index.php? ... cue_System

[rojter]

Hi johannes!

Yes the front LED turns blue after about 7-10 minutes from the time I press the power button, which is much longer than usual. It also takes 3-4 minutes for it to shut down.

There's some red blinking from the back of the B3 which I just now noticed after some 15 months of usage, is that perhaps the HDD activity LED?

[johannes]

Correct, this is HD activity.

[ryz]

Sounds like full disk to me due to many mails in root account search the forum for more info. I am on my phone and n a hurry right now.

[johannes]

you are right, it actually fits the error description better. Anyhow, you need to get access to free up space, so go for the rescue stick and have a look.

[rojter]

Hi again!

Used the rescue stick and did the following:

Code: Select all

# df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/root                 6.0M      3.9M      1.8M  68% /
tmpfs                   252.1M     44.0K    252.0M   0% /tmp
df: /mnt/usb: No such file or directory
/dev/sda1                 9.8G      1.6G      7.8G  17% /mnt

where the /dev/sda1 is the mounted root disk.

ryz & johannes, I guess that was the one you referred to as being full? If that's the case I guess this isn't the problem.

Haven't read that many logs before, but if possible I'm trying to decipher all of the logs I can find for potential hints. Do you know if there are particular logs that might be more interesting than others or other means of getting to the bottom of this?

[Gordon]

Could be a mounting issue. You could try edit fstab from the rescue system and place a hash tag in front of the /home mount. It may also be a good idea to reformat swap (`mkswap /dev/sda3`). Then retry booting.

[Ubi]

Also, the report that DHCPd is still working is misleading. The client caches a copy of the last dhcp answer for a certain grace period. This suggests the dhcp server is still up while it isnt

If you do get new dhcp answers from the b3 then the issue does not seem to be the disk, as dhcpd is started after /etc/fstab is processed. Even so, Gordons suggestion is a very good test.

[rojter]

Thank you for your suggestions Gordon. I did as you suggested but I still experience the same things. E.g. long boot time and not being able to reach the web interface. But at least being able to ping my bubba.

Code: Select all

# vi /mnt/etc/fstab
/dev/mapper/bubba-storage       #/home  ext3    defaults        0       2
/dev/sda1       /       ext3    noatime,defaults        0       1
/dev/sda3       none    swap    sw      0       0
/proc   /proc   proc    defaults        0       0
usbfs   /proc/bus/usb   usbfs   defaults        0       0

Code: Select all

# mkswap /dev/sda3
Setting up swapspace version 1, size = 1073737728 bytes

Ubi, I didn't know that. You're correct, I just made an assumption based on my limited knowledge. Thank you for your clarification. Unfortunately I'm not sure how to get a new lease. I can go through some boxes to try and find some device with an ethernet port.

To see how the disc feels I connected it to my desktop to read the SMART values of the disk with the HWiNFO tool and they were all fine:

Self-Monitoring, Analysis and Reporting Technology
Raw Read Error Rate: 196/51, Worst: 196 (Data = 6994)
Spin Up Time: 208/21, Worst: 170 (Data = 4566)
Start/Stop Count: 97/Always OK, Worst: 97 (Data = 3551)
Reallocated Sector Count: 175/140, Worst: 175 (Data = 473)
Seek Error Rate: 200/Always OK, Worst: 200
Power-On Hours/Cycle Count: 59/Always OK, Worst: 59 (Data = 30497)
Spin Retry Count: 100/Always OK, Worst: 100
Calibration Retry Count: 100/Always OK, Worst: 100
Power Cycle Count: 100/Always OK, Worst: 100 (Data = 185)
Power-Off Retract Count: 200/Always OK, Worst: 200 (Data = 46)
Load/Unload Cycle Count: 99/Always OK, Worst: 99 (Data = 303408)
Temperature 120/Always OK, Worst: 107 (Data = 30.0 °C)
Reallocation Event Count: 1/Always OK, Worst: 1 (Data = 373)
Current Pending Sector Count: 198/Always OK, Worst: 196 (Data = 792)
Off-Line Uncorrectable Sector Count: 200/Always OK, Worst: 196 (Data = 72)
UltraDMA/SATA CRC Error Rate: 200/Always OK, Worst: 200
Write/Multi-Zone Error Rate: 128/Always OK, Worst: 1 (Data = 19421)

I'll try any test you throw at me. Meanwhile I'll ponder a bit on my own. Thank you all for you help so far.

[Gordon]

The web root is on the /home mount, so yes it is very likely that this won't work. Question is: can you now ssh into the box? You could then try to manually mount /home and look for errors.

PS The hash tag should have been in front of the whole line, but apparently mount_all got so confused from where you put it that it did the job as well.

[Ubi]

Afaik you cannot SSH into a box as a regular user if the users $HOME is not available (i.e. when /home is not mounted). I think this doesnt apply to all distros, but you can circumvent by making a new /home/yourusername in the unmounted /home folder. Dont forget the chown and chmod.

Or of course, you can SSH as root, but this has to be enabled in /etc/ssh/sshd_config (allow_root_access=yes)

[Gordon]

You can. It will show an error though on account of your home directory being invalid, but you can su to root from that state.

[Ubi]

ok good to know. RHEL will reject the login that's why I thought of it.

[rojter]

My bad, I went and corrected fstab:

Code: Select all

# vi /mnt/etc/fstab
#/dev/mapper/bubba-storage       /home  ext3    defaults        0       2
/dev/sda1       /       ext3    noatime,defaults        0       1
/dev/sda3       none    swap    sw      0       0
/proc   /proc   proc    defaults        0       0
usbfs   /proc/bus/usb   usbfs   defaults        0       0

...and reformatted swap again:

Code: Select all

# mkswap /dev/sda3
Setting up swapspace version 1, size = 1073737728 bytes

Unfortunately I still cannot ssh into the box. Still getting the

Network error: Connection timed out

from PuTTY, 20 seconds after initializing the connection request.