Interesting Traces - Out of Order versus Retransmissions

Blue Bar separator


When analyzing poor network performance one of the first things I look for are retransmissions. Wireshark does a good job of flagging retransmissions but not a perfect job. Sometimes it flags a retransmission as an out of order segment. While out of order segments may cause a performance hit, the effect is typically much less than that of a retransmission. This makes it important to confirm that segments flagged as out of order are really out of order and not retransmissions.

In this example Wireshark (well tshark, but its the same thing) is reporting that segments are out of order but in reality they are retransmissions.

H:\>"C:\Program Files\Wireshark\tshark" -r trace-ooo.pcap -Y "tcp.analysis.out_of_order"
 6907   0.269793 192.168.1.21 -> 192.168.9.6  SMB [TCP Out-Of-Order] Write AndX Response, 65536 bytes
10521   0.413026 192.168.1.21 -> 192.168.9.6  SMB [TCP Out-Of-Order] Write AndX Response, 65536 bytes
12492   0.489400 192.168.1.21 -> 192.168.9.6  SMB [TCP Out-Of-Order] Write AndX Response, 65536 bytes
H:\>

Wireshark (tshark) is looking at the sequence numbers (and other things, I suppose) to determine that the TCP segments are out of order. The out of order segment has a sequence number less than the previous segment.

H:\>"C:\Program Files\Wireshark\tshark" -r trace-ooo.pcap -Y "(frame.number == 6906 || frame.number =
= 6907 || frame.number == 6908)" -T fields -E header=y -e frame.number -e ip.src -e tcp.seq  -e exper
t.message
frame.number    ip.src  tcp.seq expert.message
6906    192.168.1.21    16066
6907    192.168.1.21    13618   Out-Of-Order segment
6908    192.168.1.21    16066

H:\>"C:\Program Files\Wireshark\tshark" -r trace-ooo.pcap -Y "(frame.number == 10520 || frame.number 
== 10521 || frame.number == 10522)" -T fields -E header=y -e frame.number -e ip.src -e tcp.seq -e exp
ert.message
frame.number    ip.src  tcp.seq expert.message
10520   192.168.1.21    24583
10521   192.168.1.21    22186   Out-Of-Order segment
10522   192.168.1.21    24583

H:\>"C:\Program Files\Wireshark\tshark" -r trace-ooo.pcap -Y "(frame.number == 12491 || frame.number 
== 12492 || frame.number == 12493)" -T fields -E header=y -e frame.number -e ip.src -e tcp.seq -e exp
ert.message
frame.number    ip.src  tcp.seq expert.message
12491   192.168.1.21    29224
12492   192.168.1.21    26776   Out-Of-Order segment
12493   192.168.1.21    29224

So why do I think they are retransmissions and not out of order segments? It's the IP Identification (ip.id) field value. If the segments were out of order I would expect to see a smaller ip.id value for the segment flagged as out of order when compared to the segment immediately before it, instead I see that the ip.id value has been incremented by 1.

H:\>"C:\Program Files\Wireshark\tshark" -r trace-ooo.pcap -Y "(frame.number == 6906 || frame.number =
= 6907 || frame.number == 6908)" -T fields -E header=y -e frame.number -e ip.src -e ip.id -e tcp.seq 
-e tcp.ack -e tcp.len -e expert.message
frame.number    ip.src  ip.id   tcp.seq tcp.ack tcp.len expert.message
6906    192.168.1.21    0x35e6  16066   20689241        0
6907    192.168.1.21    0x35e7  13618   20689241        1448    Out-Of-Order segment
6908    192.168.1.21    0x35e8  16066   20692137        0

H:\>"C:\Program Files\Wireshark\tshark" -r trace-ooo.pcap -Y "(frame.number == 10520 || frame.number 
== 10521 || frame.number == 10522)" -T fields -E header=y -e frame.number -e ip.src -e ip.id -e tcp.s
eq -e tcp.ack -e tcp.len -e expert.message
frame.number    ip.src  ip.id   tcp.seq tcp.ack tcp.len expert.message
10520   192.168.1.21    0x4598  24583   31643469        0
10521   192.168.1.21    0x4599  22186   31644917        1448    Out-Of-Order segment
10522   192.168.1.21    0x459a  24583   31647813        0

H:\>"C:\Program Files\Wireshark\tshark" -r trace-ooo.pcap -Y "(frame.number == 12491 || frame.number 
== 12492 || frame.number == 12493)" -T fields -E header=y -e frame.number -e ip.src -e ip.id -e tcp.s
eq -e tcp.ack -e tcp.len -e expert.message
frame.number    ip.src  ip.id   tcp.seq tcp.ack tcp.len expert.message
12491   192.168.1.21    0x5032  29224   37612197        0
12492   192.168.1.21    0x5033  26776   37615093        1448    Out-Of-Order segment
12493   192.168.1.21    0x5034  29224   37617989        0

The increase might have been larger than 1, it depends on how the sending TCP/IP stack creates the ip.id value and on how busy the stack is. Some stacks use a different counter for every TCP connection while others use one counter for all connections; the key however is that the value was larger not smaller.

Why does Wireshark flag these segments as "out of order" and not retransmissions? I have no idea; the original segment is in the trace. You can see that the "out of order" segments are seen about 40 ms after the original but the latter segment also contains new data (based on the tcp length) and is not just a retransmission of the old data.

H:\>"C:\Program Files\Wireshark\tshark" -r trace-ooo.pcap -Y "tcp.seq == 13618 && tcp.len > 0" -T fie
lds -E header=y -e frame.number -e frame.time_delta_displayed -e ip.src -e ip.id -e tcp.seq -e tcp.ac
k -e tcp.len -e expert.message
frame.number    frame.time_delta_displayed     ip.src  ip.id   tcp.seq tcp.ack tcp.len expert.message
5855    0.000000000     192.168.1.21    0x31cb  13618   17557181        51
6907    0.040750000     192.168.1.21    0x35e7  13618   20689241        1448  Out-Of-Order segment

H:\>"C:\Program Files\Wireshark\tshark" -r trace-ooo.pcap -Y "tcp.seq == 22186 && tcp.len > 0" -T fie
lds -E header=y -e frame.number -e frame.time_delta_displayed -e ip.src -e ip.id -e tcp.seq -e tcp.ac
k -e tcp.len -e expert.message
frame.number    frame.time_delta_displayed     ip.src  ip.id   tcp.seq tcp.ack tcp.len expert.message
9514    0.000000000     192.168.1.21    0x4040  22186   28598289        51
10521   0.042885000     192.168.1.21    0x4599  22186   31644917        1448  Out-Of-Order segment

H:\>"C:\Program Files\Wireshark\tshark" -r trace-ooo.pcap -Y "tcp.seq == 26776 && tcp.len > 0" -T fie
lds -E header=y -e frame.number -e frame.time_delta_displayed -e ip.src -e ip.id -e tcp.seq -e tcp.ac
k -e tcp.len -e expert.message
frame.number    frame.time_delta_displayed     ip.src  ip.id   tcp.seq tcp.ack tcp.len expert.message
11464   0.000000000     192.168.1.21    0x4ae8  26776   34483033        51
12492   0.040456000     192.168.1.21    0x5033  26776   37615093        1448  Out-Of-Order segment

What does it look like when the packets are truely out of order? In these two examples you can see the sequence number in the out of order frame is smaller than the sequence number in the previous frame and the IP Identification value is also smaller.

H:\>"C:\Program Files\Wireshark\tshark" -r trace-ooo-2.pcap -Y "tcp.analysis.out_of_order"
5463 0.785274000 192.168.1.21 -> 10.13.16.42  TCP 0 0x7b6d (31597) [TCP Out-Of-Order] [TCP segment of
 a reassembled PDU]
5511 0.790167000 192.168.1.21 -> 10.13.16.42  TCP 0 0x7bfe (31742) [TCP Out-Of-Order] [TCP segment of
 a reassembled PDU]

H:\>"C:\Program Files\Wireshark\tshark" -r trace-ooo-2.pcap -Y "(frame.number == 5462 || frame.number
 == 5463 || frame.number == 5464)" -T fields -E header=y -e frame.number -e ip.src -e ip.id -e tcp.se
 q -e expert.message
frame.number    ip.src  ip.id   tcp.seq expert.message
5462    192.168.1.21    0x7b6e  36252907        
5463    192.168.1.21    0x7b6d  36251459        Out-Of-Order segment
5464    192.168.1.21    0x7b6f  36254355

H:\>"C:\Program Files\Wireshark\tshark" -r trace-ooo-2.pcap -Y "(frame.number == 5510 || frame.number
 == 5511 || frame.number == 5512)" -T fields -E header=y -e frame.number -e ip.src -e ip.id -e tcp.se
 q -e expert.message
frame.number    ip.src  ip.id   tcp.seq expert.message
5510    192.168.1.21    0x7bff  36460151        
5511    192.168.1.21    0x7bfe  36458703        Out-Of-Order segment
5512    192.168.1.21    0x7c00  36461599


Blue Bar separator
This page was last modified on 14-05-05
mailbox Send comments and suggestions
to noah@noahdavids.org