Network stopped working

[Gordon]

Just a thought...

You might try to boot the rescue system and put a hashtag in front of /dev/mapper/bubba-storage in fstab on /dev/sda1. Maybe also recreate swap on /dev/sda3. There may be an issue with mount-all and that would most definitely keep the system from starting the basic required services network and sshd.

[d_rylndr]

Thank you both for your help and patience.

I'll try #tagging mounts monday morning.

[d_rylndr]

...boot the rescue system and put a hashtag in front of /dev/mapper/bubba-storage in fstab on /dev/sda1. Maybe also recreate swap on /dev/sda3. There may be an issue with mount-all and that would most definitely keep the system from starting the basic required services network and sshd.

Tried this now, didn't work... #tagged /dev/mapper/bubba-storage and recreated swap (mkswap /dev/sda3)

Here's the b3's fstab:

Code: Select all

/dev/sda1       /       ext3    noatime,defaults        0       1
#/dev/mapper/bubba-storage      /home   ext3    defaults                0
/dev/sda3      none    swap    sw                      0       0
usbfs           /proc/bus/usb   usbfs   defaults        0       0
/proc           /proc   proc    defaults                0       0

So, next step?

[Gordon]

Still no network? Starting to run out of options...

Only thing that comes to mind right now is to disable services that you don't need for a clean boot - mdadm-raid, lvm2, checkfs, [mount*]

Another thing you might attempt is to work from the rescue system. Not sure if this will work, different kernel and so on...

Code: Select all

swapon /dev/sda3
mkdir /mnt/bubba
mount /dev/sda1 /mnt/bubba
# may need to copy /lib/modules over to /mnt/bubba/lib before doing the next step
chroot /mnt/bubba
. /etc/profile

You could then attempt to start boot time services and see where it starts asking you questions, provided there are no missing libraries.

[Ubi]

I think we should remember that this is a clean system. This strongly suggests the problem is with hardware or interaction with hardware. We've already seen the issue is not the NIC, or in fact the primary system, as the rescue stick boots fine. This kinda leaves the disk, which wasn't completely formatted during the reinstall, and may contain residual errors (mainly the partition table and boot sectors etc). Therefore, my suggestion would be to take a fresh new disk, reinstall on that disk with full format options, and see if the system boots up by itself. The disk need only be 20GB or so.

[Gordon]

Yes. But if the system would not boot at all it would not show a blue front light. And I may be wrong here, but I don't think the disk actually needs to be bootable to run the system. It just requires /dev/sda1 to contain /boot/uImage

[d_rylndr]

Ok, I dug out an old spare harddrive and connected the b3 to it, made a complete reinstallation with format and partition.
b3 starts up and light turns blue.
And:
nothing. Nothing at all. b3 doesn't get an IP and if I plug a client into the LAN port it doesn't even notice there is a connection.
So this suggest hardware failure, right? Just not harddrive as I use another one here.
But it does boot a rescue stick and gets an IP then. How come!?

[Ubi]

That is a very, very good question.

I suppose you are reaching the plateau of sending the thing to Excito, but if you're in for the long haul, it may actually be interesting to connect the B3 to a crosscable with a computer running wireshark, so see if the B3 is sending out DHCP requests onto either of the NICs. I'm no big expert on wireshark though.

[Gordon]

This has got to be the weirdest case yet

How can it be broken if booting normally when it is clearly not broken if booting the rescue system? I agree that a software issue defeats logic, but if it is hardware then what hardware does the regular system access that the rescue system does not? What do the logs say?

[johannes]

Hmm, now forgive me if I'm out of bounds here (only read the last page of posts), but to me this sounds as a borked USB stick? Did you try to throw the one you have far away and get a new one, re-download the installer, re-unzip and retry? Don't touch the config file, just in case you use some text editor which messes things up.

I have actually seen this exact issue a few times, without ever understanding 100% what it was, B3 booting (blue) but no network. My fix has always been to throw away the USB stick and take a new one and it worked (however, again, not 100% sure this was the issue).

[Ubi]

Erm, Johannes, the USB stick is the only one that *does* work! The thing refuses to boot normally, only from USB. Well actually it does boot into blue light, but then its inaccessible from outside

Whcih reminds me: what you CAN do is add an init script that runs after network, maybe at 99 or so, which does nothing but run ifconfig and maybe some other things like route, and output everything to a file in /root (not temp).

At least that will show if there is a network built up, and what it thinks its ip address is.

[johannes]

Erm, Johannes, the USB stick is the only one that *does* work! The thing refuses to boot normally, only from USB. Well actually it does boot into blue light, but then its inaccessible from outside

Ok, then the USB stick is probably OK but I have to persist, this does not mean that the USB stick is good to install from. There could be a file corruption in the payload (=target FS) which causes the install to be incomplete. This has nothing to do with the rescue system (which is the same system running when installing).

In fact the more I think about this the more certain I am. Which is not much, but still. I'd say take a new stick and re-try without touching any options, I'll bet it will work.

[Ubi]

I accept that bet.

[Gordon]

Hmm, now forgive me if I'm out of bounds here (only read the last page of posts), but to me this sounds as a borked USB stick? Did you try to throw the one you have far away and get a new one, re-download the installer, re-unzip and retry? Don't touch the config file, just in case you use some text editor which messes things up.

I have actually seen this exact issue a few times, without ever understanding 100% what it was, B3 booting (blue) but no network. My fix has always been to throw away the USB stick and take a new one and it worked (however, again, not 100% sure this was the issue).

That actually makes a lot of sense. Remember what I said before, that the stick does not appear to hold an OS and that the rescue system is most likely booted from firmware. The argument that the stick is the only that does work therefore does not validate the integrity of the stick itself. There's no telling what could happen if the payload file on that stick was corrupted. Well, at least not without a screen attached to the B3 and that just happens to be impossible.

[johannes]

Exactly. To clarify how stuff works:

1) Rescue stick:
Boot loader (uboot, located in tiny flash on motherboard) loads a RAM file system and kernel from USB stick, a small, stripped Linux system, and boots from that.

2) Installer:
Boot loader loads the same kernel and file system to RAM, and invokes an install script after boot, which copies kernel and file system from the payload file to disk.

And again, theory is that this payload is corrupted (can be USB flash defect, or simple file corruption), and this causes missing files on target file system.