The perils of port number reuse
This trace shows a 144 second delay in a client reconnecting to the server. Frames 1 thru 4 show a normal connection shutdown initated by the server. The client then atempts to reconnect in frame 5, the server sends a ACK not a SYN-ACK in frame 6 and the client sends an RST in frame 7. This sequence repeats until the server reponds to the SYN in frame 20 with a SYN-ACK in frame 21.
The client's RST segment in frame 7 can be understood by looking at the client's port number in the SYN segment in frame 5 and the acknowledgment number in the server's ACK segment in frame 6. The client's port number is 50964, this is the same client port number used in the connection that was shutdown in frames 1 thru 4. The server is not acknowledging the sequence number in the client's SYN segment but the sequence number in the client's FIN segment from frame 3. The client sends a reset because the response makes no sense to it. Notice that the next connection attempt in frame 8 uses the same client port number of 50964 but a different sequence number and results in an ACK from the server in frame 9 again acknowledging the FIN segment from frame 3. This pattern of SYN/wrong ACK/RST continues until the server responds with a segment acknowledging the SYN's sequence number in frame 21.
So why is the server acknowledging the wrong sequence number and what changed that it suddenly acknowldged the correct sequence number?
The problem is that the client is trying to use the 4-tuple <172.16.3.22, 192.168.48.19, 50964, 31234> while the server has that 4-tuple, created from the previous connection that was shutdown in frames 1 thru 4, in a TIME-WAIT state. The TIME-WAIT state is one of the final states of a TCP connection. A connection is shutdown with a sequence of segments FIN/ACK/FIN/ACK, that is host A sends a segment with the FIN flag set, host B acknowledges the FIN, host B sends a segment with the FIN flag set (these two segments can be combined into one segment) and finally host A acknowledges side B's FIN. At that point host A enters the TIME-WAIT state. According to RFC 793 the purpose of the TIME-WAIT state is to "...waiting for enough time to pass to be sure the remote [side B] TCP received the acknowledgment of its connection termination request [side A's final ACK segment]". The time to wait in this state is 2 times the maximum segment lifetime (MSL), the time that a segment should be able to exist on the network. The RFC defines MSL as "arbitrarily defined to be 2 minutes". Many hosts use 60 or 30 seconds but smaller values can also be configured. In this case the MSL was defined as 60 seconds. So after 120 seconds the 4-tuple in the TIME-WAIT state is removed and the identical 4-tuple can be recreated. The fact that it actually takes 144 seconds to reestabish the conection is due to the client's retry period, 3, 6, 12, 24, and 48 seconds, plus of course the 51 seconds between the initial shutdown and the first attempt to reestablish the connection.
The simplest solution is to have the client use a random port number for each connection request. If that is not possible because of program or security restrictions the server can be reconfigured to use a smaller MSL value.
No. Time Source Destination Info 1 0.000000 192.168.48.19 172.16.3.22 31234 > 50964 [FIN, ACK] Seq=1099323879 Ack=2544153492 Win=8192 Len=0 2 0.000070 172.16.3.22 192.168.48.19 50964 > 31234 [ACK] Seq=2544153492 Ack=1099323879 Win=5840 Len=0 3 0.000673 172.16.3.22 192.168.48.19 50964 > 31234 [FIN, ACK] Seq=2544153492 Ack=1099323880 Win=5840 Len=0 4 0.000677 192.168.48.19 172.16.3.22 31234 50964 [ACK] Seq=1099323880 Ack=2544153493 Win=8192 Len=0 5 51.446092 172.16.3.22 192.168.48.19 50964 > 31234 [SYN] Seq=2060930194 Win=5840 Len=0 MSS=1380 6 51.446184 192.168.48.19 172.16.3.22 31234 50964 [ACK] Seq=1099323880 Ack=2544153493 Win=8192 Len=0 7 51.446656 172.16.3.22 192.168.48.19 50964 > 31234 [RST] Seq=2544153493 Win=0 Len=0 8 54.444541 172.16.3.22 192.168.48.19 50964 > 31234 [SYN] Seq=1471229223 Win=5840 Len=0 MSS=1380 9 54.444576 192.168.48.19 172.16.3.22 31234 50964 [ACK] Seq=1099323880 Ack=2544153493 Win=8192 Len=0 10 54.445103 172.16.3.22 192.168.48.19 50964 > 31234 [RST] Seq=2544153493 Win=0 Len=0 11 60.444579 172.16.3.22 192.168.48.19 50964 > 31234 [SYN] Seq=1930046455 Win=5840 Len=0 MSS=1380 12 60.444686 192.168.48.19 172.16.3.22 31234 50964 [ACK] Seq=1099323880 Ack=2544153493 Win=8192 Len=0 13 60.445120 172.16.3.22 192.168.48.19 50964 > 31234 [RST] Seq=2544153493 Win=0 Len=0 14 72.445032 172.16.3.22 192.168.48.19 50964 > 31234 [SYN] Seq=973707112 Win=5840 Len=0 MSS=1380 15 72.445037 192.168.48.19 172.16.3.22 31234 50964 [ACK] Seq=1099323880 Ack=2544153493 Win=8192 Len=0 16 72.445512 172.16.3.22 192.168.48.19 50964 > 31234 [RST] Seq=2544153493 Win=0 Len=0 17 96.445735 172.16.3.22 192.168.48.19 50964 > 31234 [SYN] Seq=1033400944 Win=5840 Len=0 MSS=1380 18 96.445739 192.168.48.19 172.16.3.22 31234 50964 [ACK] Seq=1099323880 Ack=2544153493 Win=8192 Len=0 19 96.446216 172.16.3.22 192.168.48.19 50964 > 31234 [RST] Seq=2544153493 Win=0 Len=0 20 144.447084 172.16.3.22 192.168.48.19 50964 31234 [SYN] Seq=2672326445 Win=5840 Len=0 MSS=1380 21 144.447680 192.168.48.19 172.16.3.22 31234 > 50964 [SYN, ACK] Seq=1262544472 Ack=2672326446 Win=8192 Len=0 MSS=536
Send comments and suggestions