Checksum errors
This trace shows something I have very rarely seen, checksum errors in the higher layer protocols while the Ethernet FCS is correct. It started out with a reported problem reaching a Windows server. A quick test with ping showed a large percentage of timeouts. So the next step was a ping and a trace using wireshark.
Again the ping showed a large packet loss, 70%; but the trace only showed 50% (10 requests went out and 5 replies came back). A loss of 50% is still terrible but why does the trace show fewer packets lost then the command reported - and could that be a clue as to the overall problem?
C:\ ping 172.16.1.23 -n 10
Pinging 172.16.1.23 with 32 bytes of data:
Reply from 172.16.1.23: bytes=32 time=2ms TTL=127
Request timed out
Request timed out
Request timed out
Request timed out
Request timed out
Reply from 172.16.1.23: bytes=32 time=2ms TTL=127
Request timed out
Request timed out
Reply from 172.16.1.23: bytes=32 time=2ms TTL=127
Ping statistics for 172.16.1.23:
Packets: sent = 10, Received = 3, Lost = 7 (70% loss),
Approximate round trip times in milli-seconds:
Minimum = 0ms, Maximum = 2ms, Average = 1ms
|
No. Time Source Destination Protocol Length Info 1117 36.235130 192.168.1.150 172.16.1.23 ICMP 74 Echo (ping) request id=0x0001, seq=45/11520, ttl=128 1118 36.236785 172.16.1.23 192.168.1.150 ICMP 74 Echo (ping) reply id=0x0001, seq=45/11520, ttl=127 1147 37.234526 192.168.1.150 172.16.1.23 ICMP 74 Echo (ping) request id=0x0001, seq=46/11776, ttl=128 1309 42.007776 192.168.1.150 172.16.1.23 ICMP 74 Echo (ping) request id=0x0001, seq=47/12032, ttl=128 1429 47.008029 192.168.1.150 172.16.1.23 ICMP 74 Echo (ping) request id=0x0001, seq=48/12288, ttl=128 1605 52.007284 192.168.1.150 172.16.1.23 ICMP 74 Echo (ping) request id=0x0001, seq=49/12544, ttl=128 1806 57.007534 192.168.1.150 172.16.1.23 ICMP 74 Echo (ping) request id=0x0001, seq=50/12800, ttl=128 1935 62.007786 192.168.1.150 172.16.1.23 ICMP 74 Echo (ping) request id=0x0001, seq=51/13056, ttl=128 1936 62.009919 172.16.1.23 192.168.1.150 ICMP 74 Echo (ping) reply id=0x0001, seq=51/13056, ttl=127 1960 63.007827 192.168.1.150 172.16.1.23 ICMP 74 Echo (ping) request id=0x0001, seq=52/13312, ttl=128 1961 63.008774 172.16.1.23 192.168.1.150 ICMP 74 Echo (ping) reply id=0x0001, seq=52/13312, ttl=127 2102 68.008085 192.168.1.150 172.16.1.23 ICMP 74 Echo (ping) request id=0x0001, seq=53/13568, ttl=128 2103 68.008861 172.16.1.23 192.168.1.150 ICMP 74 Echo (ping) reply id=0x0001, seq=53/13568, ttl=127 2371 73.007375 192.168.1.150 172.16.1.23 ICMP 74 Echo (ping) request id=0x0001, seq=54/13824, ttl=128 2373 73.008166 172.16.1.23 192.168.1.150 ICMP 74 Echo (ping) reply id=0x0001, seq=54/13824, ttl=127 |
A closer look at the "missing" replies shows that the ICMP checksum was incorrect.
Frame 1961: 74 bytes on wire (592 bits), 74 bytes captured (592 bits)
Ethernet II, Src: Kti_04:d0:10 (00:00:ef:04:d0:10), Dst: AsustekC_52:18:6e (00:23:54:52:18:6e)
Internet Protocol Version 4, Src: 172.16.1.23 (172.16.1.23), Dst: 192.168.1.150 (192.168.1.150)
Internet Control Message Protocol
Type: 0 (Echo (ping) reply)
Code: 0
Checksum: 0x5527 [incorrect, should be 0x42ea]
Identifier (BE): 1 (0x0001)
Identifier (LE): 256 (0x0100)
Sequence number (BE): 52 (0x0034)
Sequence number (LE): 13312 (0x3400)
[Response To: 1960]
[Response Time: 0.947 ms]
Data (32 bytes)
0000 61 62 63 64 65 66 67 68 69 6a 6b 6c 6d 6e 6f 70 abcdefghijklmnop
0010 71 72 73 74 75 76 43 3b 06 c7 06 65 66 67 68 69 qrstuvC;...efghi
|
It also shows that the ping data bytes appear to be corrupted. Here are the last 4 ping requests and replies. Note that in frames 1961 and 2103 the bytes at offsets 0016 thru 001A do match the corresponding bytes in frames 1960 and 2102.
Frame 1935: 74 bytes on wire (592 bits), 74 bytes captured (592 bits)
Internet Control Message Protocol
Type: 8 (Echo (ping) request)
Data (32 bytes)
0000 61 62 63 64 65 66 67 68 69 6a 6b 6c 6d 6e 6f 70 abcdefghijklmnop
0010 71 72 73 74 75 76 77 61 62 63 64 65 66 67 68 69 qrstuvwabcdefghi
Frame 1936: 74 bytes on wire (592 bits), 74 bytes captured (592 bits)
Internet Control Message Protocol
Type: 0 (Echo (ping) reply)
Data (32 bytes)
0000 61 62 63 64 65 66 67 68 69 6a 6b 6c 6d 6e 6f 70 abcdefghijklmnop
0010 71 72 73 74 75 76 77 61 62 63 64 65 66 67 68 69 qrstuvwabcdefghi
Frame 1960: 74 bytes on wire (592 bits), 74 bytes captured (592 bits)
Type: 8 (Echo (ping) request)
Data (32 bytes)
0000 61 62 63 64 65 66 67 68 69 6a 6b 6c 6d 6e 6f 70 abcdefghijklmnop
0010 71 72 73 74 75 76 77 61 62 63 64 65 66 67 68 69 qrstuvwabcdefghi
Frame 1961: 74 bytes on wire (592 bits), 74 bytes captured (592 bits)
Internet Control Message Protocol
Type: 0 (Echo (ping) reply)
Checksum: 0x5527 [incorrect, should be 0x42ea]
Data (32 bytes)
0000 61 62 63 64 65 66 67 68 69 6a 6b 6c 6d 6e 6f 70 abcdefghijklmnop
0010 71 72 73 74 75 76 43 3b 06 c7 06 65 66 67 68 69 qrstuvC;...efghi
Frame 2102: 74 bytes on wire (592 bits), 74 bytes captured (592 bits)
Internet Control Message Protocol
Type: 8 (Echo (ping) request)
Data (32 bytes)
0000 61 62 63 64 65 66 67 68 69 6a 6b 6c 6d 6e 6f 70 abcdefghijklmnop
0010 71 72 73 74 75 76 77 61 62 63 64 65 66 67 68 69 qrstuvwabcdefghi
Frame 2103: 74 bytes on wire (592 bits), 74 bytes captured (592 bits)
Internet Control Message Protocol
Type: 0 (Echo (ping) reply)
Checksum: 0x5526 [incorrect, should be 0x0688]
Data (32 bytes)
0000 61 62 63 64 65 66 67 68 69 6a 6b 6c 6d 6e 6f 70 abcdefghijklmnop
0010 71 72 73 74 75 76 cf 73 3e ef 7e 65 66 67 68 69 qrstuv.s<.~efghi
Frame 2371: 74 bytes on wire (592 bits), 74 bytes captured (592 bits)
Internet Control Message Protocol
Type: 8 (Echo (ping) request)
Data (32 bytes)
0000 61 62 63 64 65 66 67 68 69 6a 6b 6c 6d 6e 6f 70 abcdefghijklmnop
0010 71 72 73 74 75 76 77 61 62 63 64 65 66 67 68 69 qrstuvwabcdefghi
Frame 2373: 74 bytes on wire (592 bits), 74 bytes captured (592 bits)
Internet Control Message Protocol
Type: 0 (Echo (ping) reply)
Data (32 bytes)
0000 61 62 63 64 65 66 67 68 69 6a 6b 6c 6d 6e 6f 70 abcdefghijklmnop
0010 71 72 73 74 75 76 77 61 62 63 64 65 66 67 68 69 qrstuvwabcdefghi
|
Rebooting the switch that the server was connected to resolved the problem, I can only assume that the some memory location in the switch was in a strange state.
I started out stating that the Ethernet FCS was correct but how do I know that? Since the host executing the ping command and running wireshark was on a different network than the Windows server the router sitting between them should have dropped any packet with an incorrect FCS also my PC's Ethernet card is just a standard card and I would expect it to drop a frame with an incorrect FCS value. I suspect that is what happened to the 5 replies that wireshark did not see.
Send comments and suggestions