Interesting Traces - Hung Linux 2.4/Read Hat 7.2 server/Connection Backlog exceeded

The following trace was taken using tcpdump on a Linux 2/4/Red Hat 7.2 system. It has been edited slightly to reduce the number of characters in each line and some spaces added so that the sequence and acknowledgement numbers line up. It shows what it looks like when connecting to a hung server application. In this case the server is listening on port 9999 and has a listen backlog of 5. The server is hung so it is no longer making calls to accept.

The first 6 connections all get established and data is sent. I have included only the 6th connection from the client (foo) remote port 2626. What it shows is that after the connection is made and data starts to be transfered the window on the server closes to 0. There after the client sends a perist probe evey so often. The complete trace shows that the probe starts off at 1 a second and the time increases to 2 minutes.

The interesting traces are next. After the 6th connection, all connection attempts are answered with a SYN-ACK like normal but if the client starts to send data the data packets are ACKed with another SYN-ACK packet, ACKing only the original SYN from the client. If you look at the client these connections are in the ESTABLISHED state but if you look on the server they are in the SYN_RCVD state. The client eventually gives up.

I've separated the trace by connection and given them colors so that it is easier to read. The first trace (black) is the 6th connection, the last to actually be completely established. The next 2 (green and blue) show the next 2 connection attempts.

15:20:37.534940 ip 62: foo.2626 > 192.168.1.50.9999: S   2155202665:2155202665(0) win 64240  (DF) (ttl 128, id 65499, len 48)

15:20:37.534940 ip 62: 192.168.1.50.9999 > foo.2626: S   38156837:38156837(0)        ack 2155202666 win 5840  (DF) (ttl 64, id 0, len 48)

15:20:37.534940 ip 60: foo.2626 > 192.168.1.50.9999: .                               ack 38156838 win 64240 (DF) (ttl 128, id 65500, len 40)

15:20:37.544940 ip 1514: foo.2626 > 192.168.1.50.9999: . 2155202666:2155204126(1460) ack 38156838 win 64240 (DF) (ttl 128, id 65501, len 1500)

15:20:37.544940 ip 54: 192.168.1.50.9999 > foo.2626: .                               ack 2155204126 win 8760 (DF) (ttl 64, id 39845, len 40)

15:20:37.544940 ip 1514: foo.2626 > 192.168.1.50.9999: . 2155204126:2155205586(1460) ack 38156838 win 64240 (DF) (ttl 128, id 65502, len 1500)

15:20:37.544940 ip 54: 192.168.1.50.9999 > foo.2626: .                               ack 2155205586 win 11680 (DF) (ttl 64, id 39846, len 40)

15:20:37.544940 ip 1514: foo.2626 > 192.168.1.50.9999: . 2155205586:2155207046(1460) ack 38156838 win 64240 (DF) (ttl 128, id 65503, len 1500)

15:20:37.544940 ip 54: 192.168.1.50.9999 > foo.2626: .                               ack 2155207046 win 14600 (DF) (ttl 64, id 39847, len 40)

  .  .   .

15:21:28.014940 ip 60: foo.2626 > 192.168.1.50.9999: . 2155271122:2155271123(1)      ack 38156838 win 64240 (DF) (ttl 128, id 27, len 41)

15:21:28.014940 ip 54: 192.168.1.50.9999 > foo.2626: .                               ack 2155271122 win 0 (DF) (ttl 64, id 39869, len 40)

15:22:20.044940 ip 60: foo.2626 > 192.168.1.50.9999: . 2155271122:2155271123(1)      ack 38156838 win 64240 (DF) (ttl 128, id 73, len 41)

15:22:20.044940 ip 54: 192.168.1.50.9999 > foo.2626: .                               ack 2155271122 win 0 (DF) (ttl 64, id 39870, len 40)

15:24:03.324940 ip 60: foo.2626 > 192.168.1.50.9999: . 2155271122:2155271123(1)      ack 38156838 win 64240 (DF) (ttl 128, id 116, len 41)

15:24:03.324940 ip 54: 192.168.1.50.9999 > foo.2626: .                               ack 2155271122 win 0 (DF) (ttl 64, id 39871, len 40)

15:26:03.344940 ip 60: foo.2626 > 192.168.1.50.9999: . 2155271122:2155271123(1)      ack 38156838 win 64240 (DF) (ttl 128, id 190, len 41)

15:26:03.344940 ip 54: 192.168.1.50.9999 > foo.2626: .                               ack 2155271122 win 0 (DF) (ttl 64, id 39872, len 40)

15:28:03.344940 ip 60: foo.2626 > 192.168.1.50.9999: . 2155271122:2155271123(1)      ack 38156838 win 64240 (DF) (ttl 128, id 234, len 41)

15:28:03.344940 ip 54: 192.168.1.50.9999 > foo.2626: .                               ack 2155271122 win 0 (DF) (ttl 64, id 39873, len 40)



15:23:07.484940 ip 62: foo.2629 > 192.168.1.50.9999: S   2192325918:2192325918(0) win 64240  (DF) (ttl 128, id 82, len 48)

15:23:07.484940 ip 62: 192.168.1.50.9999 > foo.2629: S   196158850:196158850(0)      ack 2192325919 win 5840  (DF) (ttl 64, id 0, len 48)

15:23:07.484940 ip 60: foo.2629 > 192.168.1.50.9999: .                               ack 196158851 win 64240 (DF) (ttl 128, id 83, len 40)

15:23:07.484940 ip 1514: foo.2629 > 192.168.1.50.9999: . 2192325919:2192327379(1460) ack 196158851 win 64240 (DF) (ttl 128, id 84, len 1500)

15:23:07.484940 ip 1514: foo.2629 > 192.168.1.50.9999: . 2192327379:2192328839(1460) ack 196158851 win 64240 (DF) (ttl 128, id 85, len 1500)

15:23:10.404940 ip 1514: foo.2629 > 192.168.1.50.9999: . 2192325919:2192327379(1460) ack 196158851 win 64240 (DF) (ttl 128, id 92, len 1500)

15:23:11.394940 ip 62: 192.168.1.50.9999 > foo.2629: S   196158850:196158850(0)      ack 2192325919 win 5840  (DF) (ttl 64, id 0, len 48)

15:23:11.394940 ip 60: foo.2629 > 192.168.1.50.9999: .                               ack 196158851 win 64240 (DF) (ttl 128, id 93, len 40)

15:23:16.444940 ip 1514: foo.2629 > 192.168.1.50.9999: . 2192325919:2192327379(1460) ack 196158851 win 64240 (DF) (ttl 128, id 94, len 1500)

15:23:17.394940 ip 62: 192.168.1.50.9999 > foo.2629: S   196158850:196158850(0)      ack 2192325919 win 5840  (DF) (ttl 64, id 0, len 48)

15:23:17.394940 ip 60: foo.2629 > 192.168.1.50.9999: .                               ack 196158851 win 64240 (DF) (ttl 128, id 95, len 40)

15:23:28.404940 ip 1514: foo.2629 > 192.168.1.50.9999: . 2192325919:2192327379(1460) ack 196158851 win 64240 (DF) (ttl 128, id 96, len 1500)

15:23:29.394940 ip 62: 192.168.1.50.9999 > foo.2629: S   196158850:196158850(0)      ack 2192325919 win 5840  (DF) (ttl 64, id 0, len 48)

15:23:29.394940 ip 60: foo.2629 > 192.168.1.50.9999: .                               ack 196158851 win 64240 (DF) (ttl 128, id 97, len 40)

15:23:53.064940 ip 1514: foo.2629 > 192.168.1.50.9999: . 2192325919:2192327379(1460) ack 196158851 win 64240 (DF) (ttl 128, id 112, len 1500)

15:23:53.394940 ip 62: 192.168.1.50.9999 > foo.2629: S   196158850:196158850(0)      ack 2192325919 win 5840  (DF) (ttl 64, id 0, len 48)

15:23:53.394940 ip 60: foo.2629 > 192.168.1.50.9999: .                               ack 196158851 win 64240 (DF) (ttl 128, id 113, len 40)

15:24:41.044940 ip 1514: foo.2629 > 192.168.1.50.9999: . 2192325919:2192327379(1460) ack 196158851 win 64240 (DF) (ttl 128, id 144, len 1500)

15:24:41.594940 ip 62: 192.168.1.50.9999 > foo.2629: S   196158850:196158850(0)      ack 2192325919 win 5840  (DF) (ttl 64, id 0, len 48)

15:24:41.594940 ip 60: foo.2629 > 192.168.1.50.9999: .                               ack 196158851 win 64240 (DF) (ttl 128, id 146, len 40)



15:23:41.064940 ip 62: foo.2630 > 192.168.1.50.9999: S   2200735227:2200735227(0) win 64240  (DF) (ttl 128, id 104, len 48)

15:23:41.064940 ip 62: 192.168.1.50.9999 > foo.2630: S   237148405:237148405(0)      ack 2200735228 win 5840  (DF) (ttl 64, id 0, len 48)

15:23:41.064940 ip 60: foo.2630 > 192.168.1.50.9999: .                               ack 237148406 win 64240 (DF) (ttl 128, id 105, len 40)

15:23:41.074940 ip 1514: foo.2630 > 192.168.1.50.9999: . 2200735228:2200736688(1460) ack 237148406 win 64240 (DF) (ttl 128, id 106, len 1500)

15:23:41.074940 ip 1514: foo.2630 > 192.168.1.50.9999: . 2200736688:2200738148(1460) ack 237148406 win 64240 (DF) (ttl 128, id 107, len 1500)

15:23:44.004940 ip 1514: foo.2630 > 192.168.1.50.9999: . 2200735228:2200736688(1460) ack 237148406 win 64240 (DF) (ttl 128, id 108, len 1500)

15:23:45.194940 ip 62: 192.168.1.50.9999 > foo.2630: S   237148405:237148405(0)      ack 2200735228 win 5840  (DF) (ttl 64, id 0, len 48)

15:23:45.194940 ip 60: foo.2630 > 192.168.1.50.9999: .                               ack 237148406 win 64240 (DF) (ttl 128, id 109, len 40)

15:23:50.034940 ip 1514: foo.2630 > 192.168.1.50.9999: . 2200735228:2200736688(1460) ack 237148406 win 64240 (DF) (ttl 128, id 110, len 1500)

15:23:51.194940 ip 62: 192.168.1.50.9999 > foo.2630: S   237148405:237148405(0)      ack 2200735228 win 5840  (DF) (ttl 64, id 0, len 48)

15:23:51.194940 ip 60: foo.2630 > 192.168.1.50.9999: .                               ack 237148406 win 64240 (DF) (ttl 128, id 111, len 40)

15:24:02.624940 ip 1514: foo.2630 > 192.168.1.50.9999: . 2200735228:2200736688(1460) ack 237148406 win 64240 (DF) (ttl 128, id 114, len 1500)

15:24:03.194940 ip 62: 192.168.1.50.9999 > foo.2630: S   237148405:237148405(0)      ack 2200735228 win 5840  (DF) (ttl 64, id 0, len 48)

15:24:03.194940 ip 60: foo.2630 > 192.168.1.50.9999: .                               ack 237148406 win 64240 (DF) (ttl 128, id 115, len 40)

15:24:26.664940 ip 1514: foo.2630 > 192.168.1.50.9999: . 2200735228:2200736688(1460) ack 237148406 win 64240 (DF) (ttl 128, id 130, len 1500)

15:24:27.394940 ip 62: 192.168.1.50.9999 > foo.2630: S   237148405:237148405(0)      ack 2200735228 win 5840  (DF) (ttl 64, id 0, len 48)

15:24:27.394940 ip 60: foo.2630 > 192.168.1.50.9999: .                               ack 237148406 win 64240 (DF) (ttl 128, id 132, len 40)

15:25:14.644940 ip 1514: foo.2630 > 192.168.1.50.9999: . 2200735228:2200736688(1460) ack 237148406 win 64240 (DF) (ttl 128, id 166, len 1500)

15:25:15.594940 ip 62: 192.168.1.50.9999 > foo.2630: S   237148405:237148405(0)      ack 2200735228 win 5840  (DF) (ttl 64, id 0, len 48)

15:25:15.594940 ip 60: foo.2630 > 192.168.1.50.9999: .                               ack 237148406 win 64240 (DF) (ttl 128, id 168, len 40)

This page was last modified on 03-07-13

Send comments and suggestions
to ndav1@cox.net