Interesting Traces - Every other packet failure

Blue Bar separator


I was called in to review the cause of the large number of retransmissions that the analyzer (a PC running Ethereal) was reporting. Throughput on the connection was terrible and the customer needed to improve it ASAP. The analyzer is closer to the 172.16.98.74 host but I do not believe it is on the same segment (TTL is 62 while the system's default is 64). The more remote host 192.168.32.1 is reached via a 64KB DSL link.

Sequence numbers from 172.16.98.74 in white on black indicate data that was dropped in the network, as determined by a retransmission from 172.16.98.74. Acknowledgement numbers from 172.16.98.74 in white on black indicate ACKs that were dropped in the network, as determined by a retransmission from 192.168.32.1.

As you can see every other packet from 172.16.98.74 is dropped by the network. 

1    IP-192.168.43.1   IP-172.16.98.74    350              "Src=55523   Dst=11015   .AP...   S=4139707118   L=  280   A=2970345660   W= 8192"   TCP
2    IP-172.16.98.74   IP-192.168.43.1     70   0.061628   "Src=11015   Dst=55523   .A....   S=2970345730   L=    0   A=4139707398   W=32768"   TCP
3    IP-192.168.43.1   IP-172.16.98.74    140   4.442807   "Src=55523   Dst=11015   .AP...   S=4139707398   L=   70   A=2970345660   W= 8192"   TCP
4    IP-172.16.98.74   IP-192.168.43.1     70   0.066472   "Src=11015   Dst=55523   .A....   S=2970345730   L=    0   A=4139707468   W=32768"   TCP
6    IP-192.168.43.1   IP-172.16.98.74    140   1.414085   "Src=55523   Dst=11015   .AP...   S=4139707398   L=   70   A=2970345660   W= 8192"   TCP
7    IP-172.16.98.74   IP-192.168.43.1     70   0.000551   "Src=11015   Dst=55523   .A....   S=2970345660   L=    0   A=4139707468   W=32768"   TCP
8    IP-172.16.98.74   IP-192.168.43.1    140  11.298757   "Src=11015   Dst=55523   .AP...   S=2970345660   L=   70   A=4139707468   W=32768"   TCP
10   IP-192.168.43.1   IP-172.16.98.74     64  34.029950   "Src=55523   Dst=11015   .A....   S=4139707467   L=    1   A=2970345659   W= 8192"   TCP
11   IP-172.16.98.74   IP-192.168.43.1     70   0.000538   "Src=11015   Dst=55523   .A....   S=2970345660   L=    0   A=4139707468   W=32768"   TCP
12   IP-192.168.43.1   IP-172.16.98.74    140  13.867582   "Src=55523   Dst=11015   .AP...   S=4139707468   L=   70   A=2970345660   W= 8192"   TCP
13   IP-172.16.98.74   IP-192.168.43.1     70   0.069607   "Src=11015   Dst=55523   .A....   S=2970345730   L=    0   A=4139707538   W=32768"   TCP
15   IP-192.168.43.1   IP-172.16.98.74    140   2.844203   "Src=55523   Dst=11015   .AP...   S=4139707468   L=   70   A=2970345660   W= 8192"   TCP
16   IP-172.16.98.74   IP-192.168.43.1     70   0.000292   "Src=11015   Dst=55523   .A....   S=2970345660   L=    0   A=4139707538   W=32768"   TCP
17   IP-172.16.98.74   IP-192.168.43.1    140   9.313404   "Src=11015   Dst=55523   .AP...   S=2970345660   L=   70   A=4139707538   W=32768"   TCP
19   IP-192.168.43.1   IP-172.16.98.74     64  35.834717   "Src=55523   Dst=11015   .A....   S=4139707537   L=    1   A=2970345659   W= 8192"   TCP
20   IP-172.16.98.74   IP-192.168.43.1     70   0.000496   "Src=11015   Dst=55523   .A....   S=2970345660   L=    0   A=4139707538   W=32768"   TCP
21   IP-192.168.43.1   IP-172.16.98.74    140  12.614147   "Src=55523   Dst=11015   .AP...   S=4139707538   L=   70   A=2970345660   W= 8192"   TCP
22   IP-172.16.98.74   IP-192.168.43.1     70   0.068548   "Src=11015   Dst=55523   .A....   S=2970345730   L=    0   A=4139707608   W=32768"   TCP
24   IP-192.168.43.1   IP-172.16.98.74    140   5.445152   "Src=55523   Dst=11015   .AP...   S=4139707538   L=   70   A=2970345660   W= 8192"   TCP
25   IP-172.16.98.74   IP-192.168.43.1     70   0.000396   "Src=11015   Dst=55523   .A....   S=2970345660   L=    0   A=4139707608   W=32768"   TCP
26   IP-172.16.98.74   IP-192.168.43.1    140   6.168950   "Src=11015   Dst=55523   .AP...   S=2970345660   L=   70   A=4139707608   W=32768"   TCP
28   IP-192.168.43.1   IP-172.16.98.74     64  39.631839   "Src=55523   Dst=11015   .A....   S=4139707607   L=    1   A=2970345659   W= 8192"   TCP
29   IP-172.16.98.74   IP-192.168.43.1     70   0.000541   "Src=11015   Dst=55523   .A....   S=2970345660   L=    0   A=4139707608   W=32768"   TCP
30   IP-192.168.43.1   IP-172.16.98.74    140   9.364319   "Src=55523   Dst=11015   .AP...   S=4139707608   L=   70   A=2970345660   W= 8192"   TCP
31   IP-172.16.98.74   IP-192.168.43.1     70   0.070426   "Src=11015   Dst=55523   .A....   S=2970345730   L=    0   A=4139707678   W=32768"   TCP
33   IP-192.168.43.1   IP-172.16.98.74    140   9.905233   "Src=55523   Dst=11015   .AP...   S=4139707608   L=   70   A=2970345660   W= 8192"   TCP
34   IP-172.16.98.74   IP-192.168.43.1     70   0.000525   "Src=11015   Dst=55523   .A....   S=2970345660   L=    0   A=4139707678   W=32768"   TCP
35   IP-172.16.98.74   IP-192.168.43.1    140   1.137104   "Src=11015   Dst=55523   .AP...   S=2970345660   L=   70   A=4139707678   W=32768"   TCP
37   IP-192.168.43.1   IP-172.16.98.74     64  45.450462   "Src=55523   Dst=11015   .A....   S=4139707677   L=    1   A=2970345659   W= 8192"   TCP
38   IP-172.16.98.74   IP-192.168.43.1     70   0.000434   "Src=11015   Dst=55523   .A....   S=2970345660   L=    0   A=4139707678   W=32768"   TCP
39   IP-192.168.43.1   IP-172.16.98.74    140   4.118404   "Src=55523   Dst=11015   .AP...   S=4139707678   L=   70   A=2970345660   W= 8192"   TCP
40   IP-172.16.98.74   IP-192.168.43.1     70   0.068311   "Src=11015   Dst=55523   .A....   S=2970345730   L=    0   A=4139707748   W=32768"   TCP
42   IP-172.16.98.74   IP-192.168.43.1    140  10.473988   "Src=11015   Dst=55523   .AP...   S=2970345660   L=   70   A=4139707748   W=32768"   TCP
43   IP-192.168.43.1   IP-172.16.98.74    142   0.073491   "Src=55523   Dst=11015   .AP...   S=4139707748   L=   72   A=2970345730   W= 8192"   TCP
44   IP-172.16.98.74   IP-192.168.43.1   1138   0.000018   "Src=11015   Dst=55523   .AP...   S=2970345730   L= 1068   A=4139707820   W=32768"   TCP
46   IP-172.16.98.74   IP-192.168.43.1   1138  12.669083   "Src=11015   Dst=55523   .AP...   S=2970345730   L= 1068   A=4139707820   W=32768"   TCP
47   IP-192.168.43.1   IP-172.16.98.74    142   0.207162   "Src=55523   Dst=11015   .AP...   S=4139707820   L=   72   A=2970346798   W= 7336"   TCP
48   IP-172.16.98.74   IP-192.168.43.1     70   0.062315   "Src=11015   Dst=55523   .A....   S=2970346798   L=    0   A=4139707892   W=32768"   TCP
50   IP-192.168.43.1   IP-172.16.98.74    502  29.332371   "Src=55523   Dst=11015   .AP...   S=4139707820   L=  432   A=2970346798   W= 8192"   TCP
51   IP-172.16.98.74   IP-192.168.43.1     70   0.069737   "Src=11015   Dst=55523   .A....   S=2970346798   L=    0   A=4139708252   W=32768"   TCP
52   IP-172.16.98.74   IP-192.168.43.1    140  17.282767   "Src=11015   Dst=55523   .AP...   S=2970346798   L=   70   A=4139708252   W=32768"   TCP
54   IP-172.16.98.74   IP-192.168.43.1    140  12.836219   "Src=11015   Dst=55523   .AP...   S=2970346798   L=   70   A=4139708252   W=32768"   TCP
55   IP-192.168.43.1   IP-172.16.98.74    142   0.070238   "Src=55523   Dst=11015   .AP...   S=4139708252   L=   72   A=2970346868   W= 8192"   TCP
56   IP-172.16.98.74   IP-192.168.43.1     70   0.069377   "Src=11015   Dst=55523   .A....   S=2970346868   L=    0   A=4139708324   W=32768"   TCP

The above trace was filtered to eliminate "unrelated" packets. But when you look at an unfiltered trace it is easy to see what is happening. Note that before every drop there is an ICMP destination unreachable message from 192.168.43.253. 

1    IP-192.168.43.1   IP-172.16.98.74    350              "Src=55523   Dst=11015   .AP...   S=4139707118   L=  280   A=2970345660   W= 8192"   TCP
2    IP-172.16.98.74   IP-192.168.43.1     70   0.061628   "Src=11015   Dst=55523   .A....   S=2970345730   L=    0   A=4139707398   W=32768"   TCP
3    IP-192.168.43.1   IP-172.16.98.74    140   4.442807   "Src=55523   Dst=11015   .AP...   S=4139707398   L=   70   A=2970345660   W= 8192"   TCP
4    IP-172.16.98.74   IP-192.168.43.1     70   0.066472   "Src=11015   Dst=55523   .A....   S=2970345730   L=    0   A=4139707468   W=32768"   TCP
5    IP-192.168.43.253 IP-172.16.98.74     74   0.026837   Destination unreachable: 192.168.43.1   ICMP DestUnreach
6    IP-192.168.43.1   IP-172.16.98.74    140   1.414085   "Src=55523   Dst=11015   .AP...   S=4139707398   L=   70   A=2970345660   W= 8192"   TCP
7    IP-172.16.98.74   IP-192.168.43.1     70   0.000551   "Src=11015   Dst=55523   .A....   S=2970345660   L=    0   A=4139707468   W=32768"   TCP
8    IP-172.16.98.74   IP-192.168.43.1    140  11.298757   "Src=11015   Dst=55523   .AP...   S=2970345660   L=   70   A=4139707468   W=32768"   TCP
9    IP-192.168.43.253 IP-172.16.98.74     74   0.035309   Destination unreachable: 192.168.43.1   ICMP DestUnreach
10   IP-192.168.43.1   IP-172.16.98.74     64  34.029950   "Src=55523   Dst=11015   .A....   S=4139707467   L=    1   A=2970345659   W= 8192"   TCP
11   IP-172.16.98.74   IP-192.168.43.1     70   0.000538   "Src=11015   Dst=55523   .A....   S=2970345660   L=    0   A=4139707468   W=32768"   TCP
12   IP-192.168.43.1   IP-172.16.98.74    140  13.867582   "Src=55523   Dst=11015   .AP...   S=4139707468   L=   70   A=2970345660   W= 8192"   TCP
13   IP-172.16.98.74   IP-192.168.43.1     70   0.069607   "Src=11015   Dst=55523   .A....   S=2970345730   L=    0   A=4139707538   W=32768"   TCP
14   IP-192.168.43.253 IP-172.16.98.74     74   0.027064   Destination unreachable: 192.168.43.1   ICMP DestUnreach
15   IP-192.168.43.1   IP-172.16.98.74    140   2.844203   "Src=55523   Dst=11015   .AP...   S=4139707468   L=   70   A=2970345660   W= 8192"   TCP
16   IP-172.16.98.74   IP-192.168.43.1     70   0.000292   "Src=11015   Dst=55523   .A....   S=2970345660   L=    0   A=4139707538   W=32768"   TCP
17   IP-172.16.98.74   IP-192.168.43.1    140   9.313404   "Src=11015   Dst=55523   .AP...   S=2970345660   L=   70   A=4139707538   W=32768"   TCP
18   IP-192.168.43.253 IP-172.16.98.74     74   0.035213   Destination unreachable: 192.168.43.1   ICMP DestUnreach
19   IP-192.168.43.1   IP-172.16.98.74     64  35.834717   "Src=55523   Dst=11015   .A....   S=4139707537   L=    1   A=2970345659   W= 8192"   TCP
20   IP-172.16.98.74   IP-192.168.43.1     70   0.000496   "Src=11015   Dst=55523   .A....   S=2970345660   L=    0   A=4139707538   W=32768"   TCP
21   IP-192.168.43.1   IP-172.16.98.74    140  12.614147   "Src=55523   Dst=11015   .AP...   S=4139707538   L=   70   A=2970345660   W= 8192"   TCP
22   IP-172.16.98.74   IP-192.168.43.1     70   0.068548   "Src=11015   Dst=55523   .A....   S=2970345730   L=    0   A=4139707608   W=32768"   TCP
23   IP-192.168.43.253 IP-172.16.98.74     74   0.026574   Destination unreachable: 192.168.43.1   ICMP DestUnreach
24   IP-192.168.43.1   IP-172.16.98.74    140   5.445152   "Src=55523   Dst=11015   .AP...   S=4139707538   L=   70   A=2970345660   W= 8192"   TCP
25   IP-172.16.98.74   IP-192.168.43.1     70   0.000396   "Src=11015   Dst=55523   .A....   S=2970345660   L=    0   A=4139707608   W=32768"   TCP
26   IP-172.16.98.74   IP-192.168.43.1    140   6.168950   "Src=11015   Dst=55523   .AP...   S=2970345660   L=   70   A=4139707608   W=32768"   TCP
27   IP-192.168.43.253 IP-172.16.98.74     74   0.034882   Destination unreachable: 192.168.43.1   ICMP DestUnreach
28   IP-192.168.43.1   IP-172.16.98.74     64  39.631839   "Src=55523   Dst=11015   .A....   S=4139707607   L=    1   A=2970345659   W= 8192"   TCP
29   IP-172.16.98.74   IP-192.168.43.1     70   0.000541   "Src=11015   Dst=55523   .A....   S=2970345660   L=    0   A=4139707608   W=32768"   TCP
30   IP-192.168.43.1   IP-172.16.98.74    140   9.364319   "Src=55523   Dst=11015   .AP...   S=4139707608   L=   70   A=2970345660   W= 8192"   TCP
31   IP-172.16.98.74   IP-192.168.43.1     70   0.070426   "Src=11015   Dst=55523   .A....   S=2970345730   L=    0   A=4139707678   W=32768"   TCP
32   IP-192.168.43.253 IP-172.16.98.74     74   0.026788   Destination unreachable: 192.168.43.1   ICMP DestUnreach
33   IP-192.168.43.1   IP-172.16.98.74    140   9.905233   "Src=55523   Dst=11015   .AP...   S=4139707608   L=   70   A=2970345660   W= 8192"   TCP
34   IP-172.16.98.74   IP-192.168.43.1     70   0.000525   "Src=11015   Dst=55523   .A....   S=2970345660   L=    0   A=4139707678   W=32768"   TCP
35   IP-172.16.98.74   IP-192.168.43.1    140   1.137104   "Src=11015   Dst=55523   .AP...   S=2970345660   L=   70   A=4139707678   W=32768"   TCP
36   IP-192.168.43.253 IP-172.16.98.74     74   0.035826   Destination unreachable: 192.168.43.1   ICMP DestUnreach
37   IP-192.168.43.1   IP-172.16.98.74     64  45.450462   "Src=55523   Dst=11015   .A....   S=4139707677   L=    1   A=2970345659   W= 8192"   TCP
38   IP-172.16.98.74   IP-192.168.43.1     70   0.000434   "Src=11015   Dst=55523   .A....   S=2970345660   L=    0   A=4139707678   W=32768"   TCP
39   IP-192.168.43.1   IP-172.16.98.74    140   4.118404   "Src=55523   Dst=11015   .AP...   S=4139707678   L=   70   A=2970345660   W= 8192"   TCP
40   IP-172.16.98.74   IP-192.168.43.1     70   0.068311   "Src=11015   Dst=55523   .A....   S=2970345730   L=    0   A=4139707748   W=32768"   TCP
41   IP-192.168.43.253 IP-172.16.98.74     74   0.026792   Destination unreachable: 192.168.43.1   ICMP DestUnreach
42   IP-172.16.98.74   IP-192.168.43.1    140  10.473988   "Src=11015   Dst=55523   .AP...   S=2970345660   L=   70   A=4139707748   W=32768"   TCP
43   IP-192.168.43.1   IP-172.16.98.74    142   0.073491   "Src=55523   Dst=11015   .AP...   S=4139707748   L=   72   A=2970345730   W= 8192"   TCP
44   IP-172.16.98.74   IP-192.168.43.1   1138   0.000018   "Src=11015   Dst=55523   .AP...   S=2970345730   L= 1068   A=4139707820   W=32768"   TCP
45   IP-192.168.43.253 IP-172.16.98.74     74   0.161432   Destination unreachable: 192.168.43.1   ICMP DestUnreach
46   IP-172.16.98.74   IP-192.168.43.1   1138  12.669083   "Src=11015   Dst=55523   .AP...   S=2970345730   L= 1068   A=4139707820   W=32768"   TCP
47   IP-192.168.43.1   IP-172.16.98.74    142   0.207162   "Src=55523   Dst=11015   .AP...   S=4139707820   L=   72   A=2970346798   W= 7336"   TCP
48   IP-172.16.98.74   IP-192.168.43.1     70   0.062315   "Src=11015   Dst=55523   .A....   S=2970346798   L=    0   A=4139707892   W=32768"   TCP
49   IP-192.168.43.253 IP-172.16.98.74     74   0.026991   Destination unreachable: 192.168.43.1   ICMP DestUnreach
50   IP-192.168.43.1   IP-172.16.98.74    502  29.332371   "Src=55523   Dst=11015   .AP...   S=4139707820   L=  432   A=2970346798   W= 8192"   TCP
51   IP-172.16.98.74   IP-192.168.43.1     70   0.069737   "Src=11015   Dst=55523   .A....   S=2970346798   L=    0   A=4139708252   W=32768"   TCP
52   IP-172.16.98.74   IP-192.168.43.1    140  17.282767   "Src=11015   Dst=55523   .AP...   S=2970346798   L=   70   A=4139708252   W=32768"   TCP
53   IP-192.168.43.253 IP-172.16.98.74     74   0.035411   Destination unreachable: 192.168.43.1   ICMP DestUnreach
54   IP-172.16.98.74   IP-192.168.43.1    140  12.836219   "Src=11015   Dst=55523   .AP...   S=2970346798   L=   70   A=4139708252   W=32768"   TCP
55   IP-192.168.43.1   IP-172.16.98.74    142   0.070238   "Src=55523   Dst=11015   .AP...   S=4139708252   L=   72   A=2970346868   W= 8192"   TCP
56   IP-172.16.98.74   IP-192.168.43.1     70   0.069377   "Src=11015   Dst=55523   .A....   S=2970346868   L=    0   A=4139708324   W=32768"   TCP
57   IP-192.168.43.253 IP-172.16.98.74     74   0.026787   Destination unreachable: 192.168.43.1   ICMP DestUnreach

A look inside the destination unreachable messages shows that they correspond to the previous packet - the one that was dropped by the network.

So now we know why the packets are being dropped. It still seemed strange that the router or firewall would let every other packet through. I conjectured that there are two routers/firewalls and some kind of load balacing software is alternately sending packets to them. One of them is allowing packets through while the other is not. I never did hear from the customer if this was the case or not but after discussing this with their customer (the people on the remote network) network performance improved significantly, so I assume that this is indeed what was happening.

Blue Bar separator
This page was last modified on 04-10-27
mailbox Send comments and suggestions
to ndav1@cox.net