There is a fair amount of confusion regarding how many TCP sockets STCP can really support and how to count them. This article addresses that confusion. This is for VOS users running some version of VOS 14.7, 15.1 or 15.2, 16x or 17x. The socket_count macro at the end of the article will help you determine what your current limits are and how close you are to hitting them.
The number of TCP sockets that can be created at any one time is based on several different limits and on which of these limits you hit first.
The first limit is the size of the tc_entry array. There are 16130 entries in this array and once they are all in use no more TCP sockets can be created. Until STCP changes the way that it searches its internal data structures it is not feasible to increase the size of this array, so 16130 is the absolute maximum number of TCP sockets that you can have.
The second limit is the stcp.mN clone limit. This is the stcp device that is created in the devices.tin file. In release 14.6.1 this limit was raised from 8120 to 16000, if you upgraded from a pre-14.6.1 version of VOS you may still be using a limit of 8120 or even 5120 which was the clone limit STCP originally (VOS release 14.1) shipped with. Almost every time a TCP socket is created, a clone of this device is also created. Once you have reached the clone limit, then with the exceptions noted in the next paragraph no more TCP sockets can be created – even if there are free entries in the tc_entry array.
Not every socket uses a stcp_mN clone device. When the telnetd process accepts a connection it first creates a clone but then after some processing it frees the clone. Outgoing (there are no incoming) x25_cpc connections to the NIO also do not use a stcp.mN clone. The same is true of the sockets used in incoming and outgoing OSL connections. But NOT the OSL listening sockets. All these TCP sockets still use an entry in the tcb_entry array but they do not use a stcp.mN clone. What this means is that typically the number of used tc_entries will be greater than the number of stcp.mN device clones.
If you have a clone limit below 16000 you can increase it by
Starting in release 17.1 you can use the update_device_info command to increase the clone_limit for the current boot. However unless you change the devices.tin and create a new devices.table file the clone_limit will revert to the old value after a reboot.
Note that Stratus recommends that you increase the clone limit only high enough to meet you current needs plus expected growth. There have been multiple instances of bugs in applications causing loops that allocate sockets until they hit the stcp_mN clone limit and exhaust streams memory trying to use the sockets. Setting the clone limit to a lower value reduces the chances that streams memory will be exhausted.
The next limit is the total number of devices that the system is configured to support. You can see this by displaying the value of max_local_devices$ in analyze_system. The value is a short and the default is 4000x or 16384. The current number of devices is stored in the variable configured_local_devices$. Remember that when the system boots every entry in the devices table for the module will count toward the maximum. If, when a socket is created a stcp.mN clone is also created, then configured_local_devices$ is incremented by 1. The same holds true for x25_cpc and telnetd connections which use their own clone devices. When the clone is freed the value is decremented by 1. If the number of devices defined when the system is booted is X you will not be able to create more than 16384 – X sockets.
You can raise the maximum number of devices just by patching the max_local_devices$ with the analyze_system request set_word. You can do this at any point after the system is booted (see figure 1).
analyze_system -request_line 'set_word max_local_devices$ 4e20' -quit VOS Release 14.7.2am, analyze_system Release 14.7.2am Current process is 4055, ptep C34C84C0, Noah_Davids.CAC addr from to C094250C 4000 4E20 |
The final limitation is the amount of memory available to the STREAMS subsystem. When a socket is created some amount of memory is needed. The amount used is added to the amount used by the Streams subsystem. If that amount exceeds certain thresholds you will not be able to create the socket. There are three thresholds LO, MED, HI. Socket creation checks both the HI and MED thresholds so if your Streams memory usage exceeds the MED threshold you will not be able to create a socket.
You can increase the amount of memory available to the STREAMS subsystem by changing the sys_denominator parameters with the analyze_system request set_streams_param. The default value is 8 which corresponds to 1/8 of system memory. Changing the value to 4 will double the available amount of streams memory. Like max_local_devices$, you can make this change any time after the system is booted.
analyze_system -request_line 'set_streams_param sys_denominator 4' -quit VOS Release 14.7.2am, analyze_system Release 14.7.2am Current process is 4055, ptep C34C84C0, Noah_Davids.CAC Changing sys max heap denomin (sys_denominator) from 8 to 4 |
The following macro, socket_count, displays the currently used number of entries in the tc_entry array, the number of stcp.mN clone devices and just calls netstat and counts the number of TCP sockets displayed. It also displays the number of devices and the current streams memory usage as a percentage of the LO, MED and HI thresholds. The LO threshold because if you have exceeded the LO threshold you will have problems moving data through STREAMS even if you can still create sockets.
socket_count OpenVOS Release 17.0.1aj, analyze_system Release 17.0.1aj Current process is 126, ptep 8DC4F000, Noah_Davids.CAC as: as: as: as: as: as: The number of tc_entries at 09-05-14 14:06:58 mst is 81 out of 16130 The stcp.m16 clone count at 09-05-14 14:06:58 mst is 52 out of 5120 TCP sockets from netstat at 09-05-14 14:06:58 mst is 79 The number of devices at 09-05-14 14:06:58 mst is 128 out of 16384 0.567% of LO Streams memory threshold in use as of 09-05-14 14:06:58 mst 0.52% of MED Streams memory threshold in use as of 09-05-14 14:06:58 mst 0.5% of HI Streams memory threshold in use as of 09-05-14 14:06:58 mst ready 14:06:58 |
Some of you may be asking why there is a date-time stamp on each line of output. The reason is that if you monitor your system by periodically executing this command and dumping all the output into a single file you can extract a set of counters, for example, the tc_entry count with the "display -match tc_entry" command and have the time stamps available.
d x.cm %phx_vos#m16_mas>SysAdmin>Noah_Davids>x.cm 09-05-14 15:15:49 mst &label again socket_count sleep -minutes 5 &goto again ready 15:15:49 d x.out -match tc_entries %phx_vos#m16_mas>SysAdmin>Noah_Davids>x.out 09-05-14 15:16:03 mst The number of tc_entries at 09-05-14 14:38:18 mst is 81 out of 16130 The number of tc_entries at 09-05-14 14:43:18 mst is 81 out of 16130 The number of tc_entries at 09-05-14 14:48:19 mst is 81 out of 16130 The number of tc_entries at 09-05-14 14:53:19 mst is 81 out of 16130 The number of tc_entries at 09-05-14 14:58:19 mst is 81 out of 16130 The number of tc_entries at 09-05-14 15:03:20 mst is 81 out of 16130 The number of tc_entries at 09-05-14 15:08:20 mst is 81 out of 16130 The number of tc_entries at 09-05-14 15:13:21 mst is 81 out of 16130 ready 15:16:03 |
& socket_count begins here & & socket_count.cm & version 2.0 06-11-01 & 2.1 08-02-07 changed the way tc_entries and netstat lines are & counted to fix a problem caused when the process & directory is a link & 2.2 10-11-26 added disclaimer & 2.3 13-01-17 added changes needed to correctly determine the & maximum tc_entries value on 17.1 & Noah Davids Stratus CAC noah.davids@stratus.com & & This software is provided on an "AS IS" basis, WITHOUT ANY WARRANTY OR ANY & SUPPORT OF ANY KIND. The AUTHOR SPECIFICALLY DISCLAIMS ANY IMPLIED WARRANTIES & OF MERCHANTABILITY OR FITNESS FOR ANY PARTICULAR PURPOSE. This disclaimer & applies, despite any verbal representations of any kind provided by the & author or anyone else. & &attach_input &set START (calc (length (system_name)) + 3) &set_string STCP_DEVICE stcp.(substr (current_module) &START&) & & &set RELEASE171 (index (module_info system_release) 'Release 17.1') & analyze_system ..attach_default_output (process_dir)>tc_entries match tcb; dump_tcentry ..detach_default_output &if &RELEASE171& > 0 &then &do ..attach_default_output (process_dir)>ntcpcon match max_tcp; dump_tcp_global &end &else &do ..attach_default_output (process_dir)>ntcpcon d ntcpcon &end ..detach_default_output ..attach_default_output (process_dir)>clones match clone_count; dump_dvt -name &STCP_DEVICE& match clone_limit; dump_dvt -name &STCP_DEVICE& ..detach_default_output ..attach_default_output (process_dir)>devices d configured_local_devices$ d max_local_devices$ ..detach_default_output ..attach_default_output (process_dir)>pmm match 'pmm_limit' ; dump_stream -stm_msg match 'hq_pool[0]->pmm_total_size' ; dump_stream -stm_msg match 'hq_pool[0]->pmm_allocated_size' ; dump_stream -stm_msg ..detach_default_output quit attach_default_output (process_dir)>netstat >system>stcp>command_library>netstat -numeric -all_sockets -protocol tcp detach_default_output & &if (exists (process_dir)>tc_entries2) &then delete_file (process_dir)>tc_entries2 -no_ask create_file (process_dir)>tc_entries2 -organization relative -record_size 256 copy_file (process_dir)>tc_entries (process_dir)>tc_entries2 -truncate &set records (file_info (process_dir)>tc_entries2 last_record) & display_line display_line &if &RELEASE171& > 0 &then &do &set NTCPCON (substr (contents (process_dir)>ntcpcon 1) 40 6) &end &else &do &set NTCPCON (calc (substr (contents (process_dir)>ntcpcon 1) 19 8)x) &end display_line The number of tc_entries at &+ (file_info (process_dir)>tc_entries date_used) is (calc &records& - 2) &+ out of &NTCPCON& & &set CLONE_COUNT (substr (contents (process_dir)>clones 1) 25) &set CLONE_LIMIT (substr (contents (process_dir)>clones 2) 25) display_line The &STCP_DEVICE& clone count at &+ (file_info (process_dir)>clones date_used) is &CLONE_COUNT& &+ out of &CLONE_LIMIT& & &if (exists (process_dir)>netstat2) &then delete_file (process_dir)>netstat2 -no_ask create_file (process_dir)>netstat2 -organization relative -record_size 256 copy_file (process_dir)>netstat (process_dir)>netstat2 -truncate &set records (file_info (process_dir)>netstat2 last_record) &if &records& = 0 &then display_line No info from netstat, probably no sockets available &else display_line TCP sockets from netstat at &+ (file_info (process_dir)>netstat date_used) is (calc &records& - 2) & &set CONFIG_DEVICES (calc (substr (contents (process_dir)>devices 1) 19 4)x) &set MAX_DEVICES (calc (substr (contents (process_dir)>devices 2) 19 4)x) display_line The number of devices at ' ' &+ (file_info (process_dir)>devices date_used) is &CONFIG_DEVICES& &+ out of &MAX_DEVICES& & &set ALLOC_MEM (substr (contents (process_dir)>pmm 6) 46) &set TOTAL_MEM (substr (contents (process_dir)>pmm 5) 46) &set LO_MEM (substr (contents (process_dir)>pmm 2) 41) &set MED_MEM (substr (contents (process_dir)>pmm 3) 41) &set HI_MEM (substr (contents (process_dir)>pmm 4) 41) &set LO_PER (substr (calc &ALLOC_MEM& / &LO_MEM& * 100) 1 5) &set MED_PER (substr (calc &ALLOC_MEM& / &MED_MEM& * 100) 1 5) &set HI_PER (substr (calc &ALLOC_MEM& / &HI_MEM& * 100) 1 5) display_line &LO_PER&% of LO Streams memory &+ threshold in use as of (file_info (process_dir)>pmm date_used) display_line &MED_PER&% of MED Streams memory &+ threshold in use as of (file_info (process_dir)>pmm date_used) display_line &HI_PER&% of HI Streams memory &+ threshold in use as of (file_info (process_dir)>pmm date_used) & & socket_count ends here |