Interesting Traces - Who sent the "destination unreachable, fragmentation needed" message

Blue Bar separator

The following trace was taken with Ethereal on a link between the NFS_server (192.168.100.130) and its firewall. The reason we were tracing the link is because an NFS_client (192.168.76.19) was not getting any response when it changed to a particular directory. Frame 3 is the query, frames 4 thru 9 are the response. Frame 10 is the key its a Destination unreachable fragmentation needed message. That explains why the client never got the reply. The only problem is that the message is from the NFS_client (192.168.76.19). Frame 11 is the query again.

It doesn't make sense that the NFS_client would send a destination unreachable fragmentation needed message and there is evidence that the sender of this message is NOT the NFS_Client. First, the Time to live value is 61 but in frames 3 and 11 it is 123. Second the Differentiated Services value is 0x48 but in frames 3 and 11 it's 0x00. The IP flags are also different 0x04 verus 0x00. Finally the Identification value is also suspect, 0x428c is way out of line with the value in frame 3, 0x816b or 0x81ba in frame 11.

So if the NFS_Client didn't send the message who did? The Internet Control Message packet contains the first 28 bytes from the packet that triggered this message. The first 20 bytes are the IP header and the next 8 are the UDP header (at least in this case). The Time to live value is 251. Since the Time to live value in frames 4 thru 9 are all 255 we know that the triggering packet has gone through 4 hops. The Time to live for the packets that we know came from the NFS_client (frames 3 and 11) have a value of 123. Since the NFS_Client is a PC running Windows XP we know that the starting Time to live is 128 so the packets have to go through 5 hops. (A sanity check trace on the NFS client also showed that the Time to live value in any packets that do make it from the NFS_server is 250, which confirms the 5 hop route). A traceroute from the NFS_server identifies the 4 hop to be a Notrel Contivity 1700 Secure IP Services Gateway.

There are a couple more interesting things that can be gleaned from the trace. The IP header in the ICMP message shows a Fragment offset of 512 and a Total length of 8144. Frame 4 has a Total length of 1500 and the Don't fragment flag was set. Frames 5 thru 9 were all fragments continuing frame 4. We know this because the Identification values are all the same. Frame 9 has a Fragment offset of 7400 and a Total length of 744. Adding the Total length to the fragment offset and subtracting the 20 byte IP header we have a total length for the packet that the NFS_server sent as 8124. Add in 20 bytes for another IP header and we have a length of 8144 - matching what is in the IP header in the ICMP message. Note also that the Identification value in the IP header of the ICMP message is byte swapped from the original packets (0x6d9f versus 0x9f6d). It looks like something rebuilt the original packet and then retransmitted it. However, the IP header in the ICMP packet also shows a Fragment offset of 512 so the packets have been refragmented. Note also that the Flags are 0x00 - the Don't fragment flag is NOT set so whoever rebuild the frame cleared the Don't Fragment flag. But if that is the case why was this message sent? I suspect a bug in the 1700's software.

By the way the solution was to configure the NFS_server so that it did not set the Don't fragment flag. Once that was done there were no more problems. Based on that I strongly suspect that it was the 1700 itself that rebuilt the frame and while it cleared the Don't fragment flag in the rebuilt frame it somehow "remembered" that it was set and generated an error. 

No.     Time        Source                Destination           Protocol Info
      3 2.339991    192.168.76.19         192.168.100.130       NFS      V3 READDIRPLUS Call (Reply In 4)

Frame 3 (174 bytes on wire, 96 bytes captured)
    Arrival Time: Aug 10, 2005 13:16:32.541828000
    Time delta from previous packet: 2.339460000 seconds
    Time since reference or first frame: 2.339991000 seconds
    Frame Number: 3
    Packet Length: 174 bytes
    Capture Length: 96 bytes
Ethernet II, Src: XX:XX:XX:XX:XX:ab, Dst: XX:XX:XX:XX:XX:f4
802.1q Virtual LAN
Internet Protocol, Src Addr: 192.168.76.19 (192.168.76.19), Dst Addr: 192.168.100.130 (192.168.100.130)
    Version: 4
    Header length: 20 bytes
    Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)
    Total Length: 156
    Identification: 0x81b6 (33206)
    Flags: 0x00
    Fragment offset: 0
    Time to live: 123
    Protocol: UDP (0x11)
    Header checksum: 0x69fd (correct)
    Source: 192.168.76.19 (192.168.76.19)
    Destination: 192.168.100.130 (192.168.100.130)
User Datagram Protocol, Src Port: 945 (945), Dst Port: nfsd (2049)
    Source port: 945 (945)
    Destination port: nfsd (2049)
    Length: 136
    Checksum: 0xc271
Remote Procedure Call, Type:Call XID:0x56b1b242

No.     Time        Source                Destination           Protocol Info
      4 2.343357    192.168.100.130       192.168.76.19         NFS      V3 READDIRPLUS Reply (Call In 3)

Frame 4 (1518 bytes on wire, 96 bytes captured)
    Arrival Time: Aug 10, 2005 13:16:32.545194000
    Time delta from previous packet: 0.003366000 seconds
    Time since reference or first frame: 2.343357000 seconds
    Frame Number: 4
    Packet Length: 1518 bytes
    Capture Length: 96 bytes
Ethernet II, Src: XX:XX:XX:XX:XX:f4, Dst: XX:XX:XX:XX:XX:ab
802.1q Virtual LAN
Internet Protocol, Src Addr: 192.168.100.130 (192.168.100.130), Dst Addr: 192.168.76.19 (192.168.76.19)
    Version: 4
    Header length: 20 bytes
    Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)
    Total Length: 1500
    Identification: 0x6d9f (28063)
    Flags: 0x06 (Don't Fragment) (More Fragments)
    Fragment offset: 0
    Time to live: 255
    Protocol: UDP (0x11)
    Header checksum: 0x94d3 (correct)
    Source: 192.168.100.130 (192.168.100.130)
    Destination: 192.168.76.19 (192.168.76.19)
User Datagram Protocol, Src Port: nfsd (2049), Dst Port: 945 (945)
    Source port: nfsd (2049)
    Destination port: 945 (945)
    Length: 8124
    Checksum: 0xa6fc
Remote Procedure Call, Type:Reply XID:0x56b1b242

No.     Time        Source                Destination           Protocol Info
      5 2.343358    192.168.100.130       192.168.76.19         IP       Fragmented IP protocol (proto=UDP 0x11, off=1480)

Frame 5 (1518 bytes on wire, 96 bytes captured)
    Arrival Time: Aug 10, 2005 13:16:32.545195000
    Time delta from previous packet: 0.000001000 seconds
    Time since reference or first frame: 2.343358000 seconds
    Frame Number: 5
    Packet Length: 1518 bytes
    Capture Length: 96 bytes
Ethernet II, Src: XX:XX:XX:XX:XX:f4, Dst: XX:XX:XX:XX:XX:ab
802.1q Virtual LAN
Internet Protocol, Src Addr: 192.168.100.130 (192.168.100.130), Dst Addr: 192.168.76.19 (192.168.76.19)
    Version: 4
    Header length: 20 bytes
    Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)
    Total Length: 1500
    Identification: 0x6d9f (28063)
    Flags: 0x06 (Don't Fragment) (More Fragments)
    Fragment offset: 1480
    Time to live: 255
    Protocol: UDP (0x11)
    Header checksum: 0x941a (correct)
    Source: 192.168.100.130 (192.168.100.130)
    Destination: 192.168.76.19 (192.168.76.19)

No.     Time        Source                Destination           Protocol Info
      6 2.343359    192.168.100.130       192.168.76.19         IP       Fragmented IP protocol (proto=UDP 0x11, off=2960)

Frame 6 (1518 bytes on wire, 96 bytes captured)
    Arrival Time: Aug 10, 2005 13:16:32.545196000
    Time delta from previous packet: 0.000001000 seconds
    Time since reference or first frame: 2.343359000 seconds
    Frame Number: 6
    Packet Length: 1518 bytes
    Capture Length: 96 bytes
Ethernet II, Src: XX:XX:XX:XX:XX:f4, Dst: XX:XX:XX:XX:XX:ab
802.1q Virtual LAN
Internet Protocol, Src Addr: 192.168.100.130 (192.168.100.130), Dst Addr: 192.168.76.19 (192.168.76.19)
    Version: 4
    Header length: 20 bytes
    Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)
    Total Length: 1500
    Identification: 0x6d9f (28063)
    Flags: 0x06 (Don't Fragment) (More Fragments)
    Fragment offset: 2960
    Time to live: 255
    Protocol: UDP (0x11)
    Header checksum: 0x9361 (correct)
    Source: 192.168.100.130 (192.168.100.130)
    Destination: 192.168.76.19 (192.168.76.19)

No.     Time        Source                Destination           Protocol Info
      7 2.343360    192.168.100.130       192.168.76.19         IP       Fragmented IP protocol (proto=UDP 0x11, off=4440)

Frame 7 (1518 bytes on wire, 96 bytes captured)
    Arrival Time: Aug 10, 2005 13:16:32.545197000
    Time delta from previous packet: 0.000001000 seconds
    Time since reference or first frame: 2.343360000 seconds
    Frame Number: 7
    Packet Length: 1518 bytes
    Capture Length: 96 bytes
Ethernet II, Src: XX:XX:XX:XX:XX:f4, Dst: XX:XX:XX:XX:XX:ab
802.1q Virtual LAN
Internet Protocol, Src Addr: 192.168.100.130 (192.168.100.130), Dst Addr: 192.168.76.19 (192.168.76.19)
    Version: 4
    Header length: 20 bytes
    Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)
    Total Length: 1500
    Identification: 0x6d9f (28063)
    Flags: 0x06 (Don't Fragment) (More Fragments)
    Fragment offset: 4440
    Time to live: 255
    Protocol: UDP (0x11)
    Header checksum: 0x92a8 (correct)
    Source: 192.168.100.130 (192.168.100.130)
    Destination: 192.168.76.19 (192.168.76.19)

No.     Time        Source                Destination           Protocol Info
      8 2.343361    192.168.100.130       192.168.76.19         IP       Fragmented IP protocol (proto=UDP 0x11, off=5920)

Frame 8 (1518 bytes on wire, 96 bytes captured)
    Arrival Time: Aug 10, 2005 13:16:32.545198000
    Time delta from previous packet: 0.000001000 seconds
    Time since reference or first frame: 2.343361000 seconds
    Frame Number: 8
    Packet Length: 1518 bytes
    Capture Length: 96 bytes
Ethernet II, Src: XX:XX:XX:XX:XX:f4, Dst: XX:XX:XX:XX:XX:ab
802.1q Virtual LAN
Internet Protocol, Src Addr: 192.168.100.130 (192.168.100.130), Dst Addr: 192.168.76.19 (192.168.76.19)
    Version: 4
    Header length: 20 bytes
    Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)
    Total Length: 1500
    Identification: 0x6d9f (28063)
    Flags: 0x06 (Don't Fragment) (More Fragments)
    Fragment offset: 5920
    Time to live: 255
    Protocol: UDP (0x11)
    Header checksum: 0x91ef (correct)
    Source: 192.168.100.130 (192.168.100.130)
    Destination: 192.168.76.19 (192.168.76.19)

No.     Time        Source                Destination           Protocol Info
      9 2.343361    192.168.100.130       192.168.76.19         IP       Fragmented IP protocol (proto=UDP 0x11, off=7400)

Frame 9 (762 bytes on wire, 96 bytes captured)
    Arrival Time: Aug 10, 2005 13:16:32.545198000
    Time delta from previous packet: 0.000000000 seconds
    Time since reference or first frame: 2.343361000 seconds
    Frame Number: 9
    Packet Length: 762 bytes
    Capture Length: 96 bytes
Ethernet II, Src: XX:XX:XX:XX:XX:f4, Dst: XX:XX:XX:XX:XX:ab
802.1q Virtual LAN
Internet Protocol, Src Addr: 192.168.100.130 (192.168.100.130), Dst Addr: 192.168.76.19 (192.168.76.19)
    Version: 4
    Header length: 20 bytes
    Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)
    Total Length: 744
    Identification: 0x6d9f (28063)
    Flags: 0x04 (Don't Fragment)
    Fragment offset: 7400
    Time to live: 255
    Protocol: UDP (0x11)
    Header checksum: 0xb42a (correct)
    Source: 192.168.100.130 (192.168.100.130)
    Destination: 192.168.76.19 (192.168.76.19)

No.     Time        Source                Destination           Protocol Info
     10 2.466501    192.168.76.19         192.168.100.130       ICMP     Destination unreachable

Frame 10 (74 bytes on wire, 74 bytes captured)
    Arrival Time: Aug 10, 2005 13:16:32.668338000
    Time delta from previous packet: 0.123140000 seconds
    Time since reference or first frame: 2.466501000 seconds
    Frame Number: 10
    Packet Length: 74 bytes
    Capture Length: 74 bytes
Ethernet II, Src: XX:XX:XX:XX:XX:ab, Dst: XX:XX:XX:XX:XX:f4
802.1q Virtual LAN
Internet Protocol, Src Addr: 192.168.76.19 (192.168.76.19), Dst Addr: 192.168.100.130 (192.168.100.130)
    Version: 4
    Header length: 20 bytes
    Differentiated Services Field: 0x48 (DSCP 0x12: Assured Forwarding 21; ECN: 0x00)
    Total Length: 56
    Identification: 0x428c (17036)
    Flags: 0x04 (Don't Fragment)
    Fragment offset: 0
    Time to live: 61
    Protocol: ICMP (0x01)
    Header checksum: 0xa753 (correct)
    Source: 192.168.76.19 (192.168.76.19)
    Destination: 192.168.100.130 (192.168.100.130)
Internet Control Message Protocol
    Type: 3 (Destination unreachable)
    Code: 4 (Fragmentation needed)
    Checksum: 0x330b (correct)
    MTU of next hop: 1400
    Internet Protocol, Src Addr: 192.168.100.130 (192.168.100.130), Dst Addr: 192.168.76.19 (192.168.76.19)
        Version: 4
        Header length: 20 bytes
        Differentiated Services Field: 0x02 (DSCP 0x00: Default; ECN: 0x02)
        Total Length: 8144
        Identification: 0x9f6d (40813)
        Flags: 0x00
        Fragment offset: 512
        Time to live: 251
        Protocol: UDP (0x11)
        Header checksum: 0x9edd (incorrect, should be 0xaccf)
        Source: 192.168.100.130 (192.168.100.130)
        Destination: 192.168.76.19 (192.168.76.19)
    Data (8 bytes)

0000  08 01 03 b1 1f bc a6 fc                           ........

No.     Time        Source                Destination           Protocol Info
     11 3.274250    192.168.76.19         192.168.100.130       NFS      [RPC retransmission of #3]V3 READDIRPLUS Call (Reply In 4)

Frame 11 (174 bytes on wire, 96 bytes captured)
    Arrival Time: Aug 10, 2005 13:16:33.476087000
    Time delta from previous packet: 0.807749000 seconds
    Time since reference or first frame: 3.274250000 seconds
    Frame Number: 11
    Packet Length: 174 bytes
    Capture Length: 96 bytes
Ethernet II, Src: XX:XX:XX:XX:XX:ab, Dst: XX:XX:XX:XX:XX:f4
802.1q Virtual LAN
Internet Protocol, Src Addr: 192.168.76.19 (192.168.76.19), Dst Addr: 192.168.100.130 (192.168.100.130)
    Version: 4
    Header length: 20 bytes
    Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)
    Total Length: 156
    Identification: 0x81ba (33210)
    Flags: 0x00
    Fragment offset: 0
    Time to live: 123
    Protocol: UDP (0x11)
    Header checksum: 0x69f9 (correct)
    Source: 192.168.76.19 (192.168.76.19)
    Destination: 192.168.100.130 (192.168.100.130)
User Datagram Protocol, Src Port: 945 (945), Dst Port: nfsd (2049)
    Source port: 945 (945)
    Destination port: nfsd (2049)
    Length: 136
    Checksum: 0xc271
Remote Procedure Call, Type:Call XID:0x56b1b242


Blue Bar separator
This page was last modified on 05-08-25
mailbox Send comments and suggestions
to ndav1@cox.net