forum.excito.org

I've been trying to track down a problem I've been having with logging in via ssh to my bubba2 and have now gotten something really weird...

I have my bubba 2 connected via it's LAN port to a separate ADSL modem/router/wireless AP. I also have a separate Ubuntu desktop. Both machines are configured to be on a subnet of the 10.x.x.x private address space as is the inward side of the router.

I can connect from the desktop machine to all external websites and ssh servers and anything else that I expect to be able to reach. If I set a suitable port forward rule I can connect inbound from the internet to the desktops ssh server.

I can connect from the bubba2 (by first ssh'ing into it from the desktop) to all things that I can connect from the desktop but from some of them I get a hang without receiving any data and likewise, if I set a suitable inbound rule on the router can log into it from some but not all internet machines that I have access to. Looking at the sshd logs, again the problematic hosts are opening a connection but no data gets through...

I've tried deleting the iptables firewall (after a certain amount of LARTting myself by not changing the policy on the INPUT chain before flushing it ....thanks for making sure the power button does a clean shutdown ).

I've tried stopping the WAN port from being configured (not entirely sure why I thought that might help).

I'm using wget to check for http data reception on both machines.

I've not been able to figure out a rule for which extneral machines work and which don't, but it might be related to some level of firewalls at the far end (physically the 'bad' machines are all on the same network segment which unfortunately happens to be my lab ) but regardless of this I'm suspecting the bubba2 since my Ubuntu desktop does everything as expected. I'd suspect the router was doing something stupid, except that it works for some hosts and everything works on the desktop.

That leaves me suspecting something in the bubba's kernel that isn't liking some aspect of that incoming packets following the initial connection but I'm an experimental physicist and not a kernel hacker .

Any thoughts of where I should be looking ?

Why not just use the WAN port?

Eek wrote:Why not just use the WAN port?

Err, because I already have a functional router and firewall and I have more than one computer with a wired connection so I need to a switch on my internal network which my existing router etc provides - I'm just using the bubba2 as a server and not using its routing capability at all.

In any case, I'm not sure that using the WAN port will help (though I guess I could try to swap the LAN and WAN port configs around). The problem is that the bubba2 can open connections but doesn't receive data from some and only some internet machines, while my other computers behind the same router/firewall have no problems.

I you have an ssh connection on the B2
what sites are working
and what sites aren't ?
are you using wget/ping/telnet to check connectivity?
cheers
Eek

Eek wrote:I you have an ssh connection on the B2
what sites are working
and what sites aren't ?
are you using wget/ping/telnet to check connectivity?
cheers
Eek

Ok quasar is my bubba2, nova is my Ubuntu desktop:

gb119@quasar:~$ wget http://www.leeds.ac.uk
--18:12:23-- http://www.leeds.ac.uk/
=> `index.html.1'
Resolving www.leeds.ac.uk... 129.11.21.9
Connecting to www.leeds.ac.uk|129.11.21.9|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 6,395 (6.2K) [text/html]

100%[============================================================================>] 6,395 --.--K/s

18:12:24 (30.80 MB/s) - `index.html.1' saved [6395/6395]

gb119@nova:~$ wget www.leeds.ac.uk
--19:16:09-- http://www.leeds.ac.uk/
=> `index.html.1'
Resolving www.leeds.ac.uk... 129.11.21.9
Connecting to www.leeds.ac.uk|129.11.21.9|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 6,395 (6.2K) [text/html]

100%[============================================================================>] 6,395 --.--K/s

19:16:09 (234.23 KB/s) - `index.html.1' saved [6395/6395]

So both machines get to www.leeds.ac.uk ok and download fine
gb119@quasar:~$ wget www.stoner.leeds.ac.uk
--18:13:49-- http://www.stoner.leeds.ac.uk/
=> `index.html.2'
Resolving www.stoner.leeds.ac.uk... 129.11.70.143
Connecting to www.stoner.leeds.ac.uk|129.11.70.143|:80... connected.
HTTP request sent, awaiting response... No data received.
Retrying.

[and goes around and around timeing out each time]

gb119@nova:~$ wget www.stoner.leeds.ac.uk
--19:17:07-- http://www.stoner.leeds.ac.uk/
=> `index.html.2'
Resolving www.stoner.leeds.ac.uk... 129.11.70.143
Connecting to www.stoner.leeds.ac.uk|129.11.70.143|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 533 [text/html]

100%[============================================================================>] 533 --.--K/s

19:17:07 (42.31 MB/s) - `index.html.2' saved [533/533]

So only nova - the Ubunut desktop - gets the data from www.stoner.leeds.ac.uk, but the bubba2 server connects but no data.

Going the other way, the only thing I've got open is the ssh port, so with sshd on the bubba2 logging in debug2 and connecting from two remote ssh clients (stonerlab.leeds.ac.uk is in the same machine room as www.stoner.leeds.ac.uk but isn't actually the same machine, dmg-stairs.msm.cam.ac.uk is in a different University:

With port forwarding on my router to nova - the Ubunut dekstop - both connect normally. Going to quasar - the bubba2 machine I get:

(tail -f /var/log/auth.log)

Oct 5 18:57:59 quasar sshd[8894]: (pam_unix) session closed for user gb119
Oct 5 18:57:59 quasar sshd[8894]: Closing connection to 131.111.103.65
Oct 5 18:57:59 quasar sshd[8894]: debug1: PAM: cleanup
Oct 5 18:58:17 quasar sshd[8913]: debug1: rexec start in 4 out 4 newsock 4 pipe 6 sock 7
Oct 5 18:58:17 quasar sshd[2358]: debug1: Forked child 8913.
Oct 5 18:58:18 quasar sshd[8913]: debug1: inetd sockets after dupping: 3, 3
Oct 5 18:58:18 quasar sshd[8913]: Connection from 131.111.103.65 port 57512
Oct 5 18:58:18 quasar sshd[8913]: debug1: Client protocol version 2.0; client software version OpenSSH_4.6
Oct 5 18:58:18 quasar sshd[8913]: debug1: match: OpenSSH_4.6 pat OpenSSH*
Oct 5 18:58:18 quasar sshd[8913]: debug1: Enabling compatibility mode for protocol 2.0
Oct 5 18:58:18 quasar sshd[8913]: debug1: Local version string SSH-2.0-OpenSSH_4.3p2 Debian-9etch2ex2
Oct 5 18:58:18 quasar sshd[8913]: debug2: fd 3 setting O_NONBLOCK
Oct 5 18:58:18 quasar sshd[8913]: debug2: Network child is on pid 8914
Oct 5 18:58:18 quasar sshd[8913]: debug2: monitor_read: 0 used once, disabling now
Oct 5 18:58:18 quasar sshd[8913]: debug2: monitor_read: 5 used once, disabling now
Oct 5 18:58:18 quasar sshd[8913]: debug2: monitor_read: 7 used once, disabling now
Oct 5 18:58:18 quasar sshd[8913]: debug1: PAM: initializing for "gb119"

[and then goes on to complete login]

Oct 5 19:01:01 quasar sshd[8941]: debug1: rexec start in 4 out 4 newsock 4 pipe 6 sock 7
Oct 5 19:01:01 quasar sshd[2358]: debug1: Forked child 8941.
Oct 5 19:01:01 quasar sshd[8941]: debug1: inetd sockets after dupping: 3, 3
Oct 5 19:01:01 quasar sshd[8941]: Connection from 129.11.69.26 port 39908
Oct 5 19:03:01 quasar sshd[8941]: fatal: Timeout before authentication for 129.11.69.26

and gets stuck.

I've also just run tcpdump as I try to make connections from these two hosts to quasar:

quasar:~# tcpdump src stonerlab.leeds.ac.uk or dst stonerlab.leeds.ac.uk
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 96 bytes
19:17:09.595148 IP stonerlab.leeds.ac.uk.39912 > quasar.ssh: S 1492739419:1492739419(0) win 5840 <mss 1380,sackOK,timestamp 857322023 0,nop,wscale 2>
19:17:09.595394 IP quasar.ssh > stonerlab.leeds.ac.uk.39912: S 1143338201:1143338201(0) ack 1492739420 win 5792 <mss 1460,sackOK,timestamp 20394856 857322023,nop,wscale 4>
19:17:09.633009 IP stonerlab.leeds.ac.uk.39912 > quasar.ssh: . ack 1 win 1460 <nop,nop,timestamp 857322064 20394856>
19:17:09.710444 IP quasar.ssh > stonerlab.leeds.ac.uk.39912: P 1:40(39) ack 1 win 362 <nop,nop,timestamp 20394885 857322064>
19:17:09.947798 IP quasar.ssh > stonerlab.leeds.ac.uk.39912: P 1:40(39) ack 1 win 362 <nop,nop,timestamp 20394945 857322064>
19:17:10.427803 IP quasar.ssh > stonerlab.leeds.ac.uk.39912: P 1:40(39) ack 1 win 362 <nop,nop,timestamp 20395065 857322064>
19:17:11.387806 IP quasar.ssh > stonerlab.leeds.ac.uk.39912: P 1:40(39) ack 1 win 362 <nop,nop,timestamp 20395305 857322064>
19:17:13.307810 IP quasar.ssh > stonerlab.leeds.ac.uk.39912: P 1:40(39) ack 1 win 362 <nop,nop,timestamp 20395785 857322064>
19:17:17.147819 IP quasar.ssh > stonerlab.leeds.ac.uk.39912: P 1:40(39) ack 1 win 362 <nop,nop,timestamp 20396745 857322064>
19:17:24.827808 IP quasar.ssh > stonerlab.leeds.ac.uk.39912: P 1:40(39) ack 1 win 362 <nop,nop,timestamp 20398665 857322 [and repeats]

quasar:~# tcpdump src dmg-stairs.msm.cam.ac.uk or dst dmg-stairs.msm.cam.ac.uk
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 96 bytes
19:15:10.630969 IP dmg-stairs.msm.cam.ac.uk.43357 > quasar.ssh: S 3830480588:3830480588(0) win 5840 <mss 1452,sackOK,timestamp 1557466252 0,nop,wscale 5>
19:15:10.631222 IP quasar.ssh > dmg-stairs.msm.cam.ac.uk.43357: S 3588963758:3588963758(0) ack 3830480589 win 5792 <mss 1460,sackOK,timestamp 20365115 1557466252,nop,wscale 4>
19:15:10.714406 IP dmg-stairs.msm.cam.ac.uk.43357 > quasar.ssh: . ack 1 win 183 <nop,nop,timestamp 1557466274 20365115>
19:15:10.789877 IP quasar.ssh > dmg-stairs.msm.cam.ac.uk.43357: P 1:40(39) ack 1 win 362 <nop,nop,timestamp 20365155 1557466274>
19:15:10.873525 IP dmg-stairs.msm.cam.ac.uk.43357 > quasar.ssh: . ack 40 win 183 <nop,nop,timestamp 1557466314 20365155>
19:15:10.873899 IP dmg-stairs.msm.cam.ac.uk.43357 > quasar.ssh: P 1:21(20) ack 40 win 183 <nop,nop,timestamp 1557466314 20365155>
19:15:10.874971 IP quasar.ssh > dmg-stairs.msm.cam.ac.uk.43357: . ack 21 win 362 <nop,nop,timestamp 20365176 1557466314>
19:15:10.895830 IP quasar.ssh > dmg-stairs.msm.cam.ac.uk.43357: P 40:744(704) ack 21 win 362 <nop,nop,timestamp 20365182 1557466314>
19:15:10.969088 IP dmg-stairs.msm.cam.ac.uk.43357 > quasar.ssh: P 21:773(752) ack 40 win 183 <nop,nop,timestamp 1557466335 20365176>
19:15:11.007799 IP quasar.ssh > dmg-stairs.msm.cam.ac.uk.43357: . ack 773 win 456 <nop,nop,timestamp 20365210 1557466335>

If I'm reading these correctly then it looks like after the initial handshake, my end keeps sending packets to the far end but doesn't get a reply packet back. Which would suggest its something funny with the routers at the far end, except that it all works on my Ubunut desktop, so something must be different between the bubba2 and the Ubunut dekstop that is falling over with whatever the far end's routing/switching is doing).

Turns out that this is another manifestation of the silicon bug in the freescale processor as described in this thread The same workaround using ethtool got the bubba2 working ok.

Probably this should be a FAQ for all instances of networking not working in really strange ways on Bubba2's.

yeah
good to know!

gb119 wrote:Turns out that this is another manifestation of the silicon bug in the freescale processor as described in this thread The same workaround using ethtool got the bubba2 working ok.

Probably this should be a FAQ for all instances of networking not working in really strange ways on Bubba2's.

Was just going to suggest that it seems like the bug; And we are planning to release a workaround for that problem in the next days or so. After that we will try to see if we can fix the problem.

/Carl