Interesting Traces - ARP Thrashing

Blue Bar separator

The following traces were taken with Ethereal. The first trace shows TCP packets between 10.17.40.196 and 10.17.36.4 and ARP packets from 10.17.40.196 with a target of 10.17.36.4 and the replies. two things make this an interesting trace. First, is the time between ARP packets. The first ARP is at 0 seconds the next ones are at 1.252, 2.504, 18.807, 35.401, 35.431 and 35.591. Second the arps at 1.252 and 2.504 do not have any preceeding TCP traffic - as if 10.17.40.196 did not see the previous ARP reply. The second trace shows packets between 10.17.40.196 and 10.17.64.117 and the same ARP query/reply pattern. While the resident time in the ARP cache is OS dependent a time of at least several minutes is typical, a resident time of less than 30 seconds is very atypical. See The ARP cache for information on ARP cache sizes.

The reason for the large number of ARPs (15% of the total number of packets in the entire trace) is ARP thrashing. 10.17.40.196 was using a 16 bit subnet mask and communicating with over 3000 hosts all on the 10.17 subnet, the ARP table was much smaller than this. To make matters worse each host sends a message to 10.17.40.196 at least once every 18 seconds. The large number of ARP requests created a situation where the 10.17.40.196 was dropping the ARP replies which just made matters worse.

The solution in this case was easy. You can see from the ARP replies that neither 10.17.36.4 nor the 10.17.64.117 was not itself answering the ARP queries. Instead 10.17.40.125 was. The MAC address shows that it is a Cisco router (OUI == 00:00:0C). The router was answering all ARP requests for anything on the 10.17.0.0 subnet (expect for hosts on 10.17.40.0). The solution was to change the subnet mask of 10.17.40.196 to have 24 bits and add a 10.17.0.0 route pointing to 10.17.40.125.

No.     Time        Source           Destination      Protocol Info
  31392 *REF*       10.17.40.196     Broadcast        ARP      Who has 10.17.36.4?  Tell 10.17.40.196
  31393 0.000000    10.17.40.125     10.17.40.196     ARP      10.17.36.4 is at 00:00:0c:05:f3:10
  31766 1.252000    10.17.40.196     Broadcast        ARP      Who has 10.17.36.4?  Tell 10.17.40.196
  31768 1.252000    10.17.40.125     10.17.40.196     ARP      10.17.36.4 is at 00:00:0c:05:f3:10
  32154 2.504000    10.17.40.196     Broadcast        ARP      Who has 10.17.36.4?  Tell 10.17.40.196
  32155 2.504000    10.17.40.125     10.17.40.196     ARP      10.17.36.4 is at 00:00:0c:05:f3:10
  32156 2.504000    10.17.40.196     10.17.36.4       TCP      59846 > 17071 [PSH, ACK]
  36446 16.654001   10.17.40.196     10.17.36.4       TCP      59846 > 17071 [ACK]
  37194 18.797001   10.17.36.4       10.17.40.196     TCP      17071 > 59846 [PSH, ACK]
  37196 18.807001   10.17.40.196     Broadcast        ARP      Who has 10.17.36.4?  Tell 10.17.40.196
  37197 18.807001   10.17.40.125     10.17.40.196     ARP      10.17.36.4 is at 00:00:0c:05:f3:10
  37198 18.807001   10.17.40.196     10.17.36.4       TCP      59846 > 17071 [PSH, ACK]
  41513 31.926000   10.17.40.196     10.17.36.4       TCP      59869 > 17051 [PSH, ACK]
  41526 31.956000   10.17.36.4       10.17.40.196     TCP      17051 > 59869 [PSH, ACK]
  42592 35.391001   10.17.36.4       10.17.40.196     TCP      17051 > 59869 [PSH, ACK]
  42604 35.401001   10.17.40.196     Broadcast        ARP      Who has 10.17.36.4?  Tell 10.17.40.196
  42606 35.411001   10.17.40.125     10.17.40.196     ARP      10.17.36.4 is at 00:00:0c:05:f3:10
  42611 35.431000   10.17.36.4       10.17.40.196     TCP      17071 > 59846 [PSH, ACK]
  42612 35.431000   10.17.36.4       10.17.40.196     TCP      17051 > 59869 [PSH, ACK]
  42615 35.431000   10.17.40.196     Broadcast        ARP      Who has 10.17.36.4?  Tell 10.17.40.196
  42616 35.431000   10.17.40.125     10.17.40.196     ARP      10.17.36.4 is at 00:00:0c:05:f3:10
  42650 35.591000   10.17.36.4       10.17.40.196     TCP      17051 > 59869 [PSH, ACK]
  42651 35.591000   10.17.40.196     Broadcast        ARP      Who has 10.17.36.4?  Tell 10.17.40.196
  42652 35.591000   10.17.40.125     10.17.40.196     ARP      10.17.36.4 is at 00:00:0c:05:f3:10
  42654 35.591000   10.17.40.196     10.17.36.4       TCP      59869 > 17051 [ACK]
  42658 35.601000   10.17.40.196     10.17.36.4       TCP      59846 > 17071 [ACK]
No.     Time        Source           Destination      Protocol Info
  36722 0.000000    10.17.40.196     Broadcast        ARP      Who has 10.17.64.117?  Tell 10.17.40.196
  36723 0.000000    10.17.40.125     10.17.40.196     ARP      10.17.64.117 is at 00:00:0c:05:f3:10
  36868 0.390000    10.17.40.196     Broadcast        ARP      Who has 10.17.64.117?  Tell 10.17.40.196
  36872 0.390000    10.17.40.125     10.17.40.196     ARP      10.17.64.117 is at 00:00:0c:05:f3:10
  36979 0.721001    10.17.40.196     10.17.64.117     TCP      57463 > 1108 [ACK]
  37004 0.781001    10.17.40.196     10.17.64.117     TCP      57463 > 1108 [ACK]
  37017 0.801000    10.17.40.196     10.17.64.117     TCP      58946 > 1106 [ACK]
  37111 1.182001    10.17.40.196     10.17.64.117     TCP      57463 > 1108 [ACK]
  37585 2.774001    10.17.40.196     10.17.64.117     TCP      57463 > 1108 [ACK]
  37650 2.974000    10.17.40.196     10.17.64.117     TCP      57463 > 1108 [ACK]
  37996 4.176000    10.17.40.196     10.17.64.117     TCP      57463 > 1108 [ACK]
  38125 4.566000    10.17.40.196     10.17.64.117     TCP      57463 > 1108 [ACK]
  40813 12.758000   10.17.40.196     10.17.64.117     TCP      57463 > 1108 [ACK]
  40897 12.958001   10.17.40.196     10.17.64.117     TCP      57463 > 1108 [ACK]
  43335 20.740000   10.17.40.196     Broadcast        ARP      Who has 10.17.64.117?  Tell 10.17.40.196
  43339 20.740000   10.17.40.125     10.17.40.196     ARP      10.17.64.117 is at 00:00:0c:05:f3:10
  43426 21.050001   10.17.40.196     Broadcast        ARP      Who has 10.17.64.117?  Tell 10.17.40.196
  43427 21.050001   10.17.40.125     10.17.40.196     ARP      10.17.64.117 is at 00:00:0c:05:f3:10
  43548 21.451000   10.17.40.196     Broadcast        ARP      Who has 10.17.64.117?  Tell 10.17.40.196
  43549 21.451000   10.17.40.125     10.17.40.196     ARP      10.17.64.117 is at 00:00:0c:05:f3:10
  43788 22.162001   10.17.40.196     10.17.64.117     TCP      57463 > 1108 [ACK]

Blue Bar separator
This page was last modified on 05-12-02
mailbox Send comments and suggestions
to ndav1@cox.net