A Perl script to calculate delta values between multiple sets of numbers
This Perl script calculates the delta values between 2 or more sets of numbers contained in a file. Each set of numbers must have the same number of lines and begin with some unique string that appears nowhere else in the file except at the beginning of the set. The numerical values may be in decimal or hexadecimal formats but all values must be in the same format. Each line in the set can contain multiple numbers and the numbers can be separated by non-numbers, i.e. labels. The output can be in 3 formats, comparison (showing current, previous and delta lines(figure 2)), deltas only (figure 3) or in a comma separated values format suitable for input into a spreadsheet (figure 4).
Usage
perl delta.pl -inputFile PATH -uniqueString STRING -numberLines NUMBER [-csvFormat] [-deltasOnly] [-hex] [-separator] [-outputFile PATH] [-loose]
-inputFile PATH | -i PATH
The path of the input file containing the sets of numbers
-uniqueString STRING | -u STRING
A unique string that identifies the start of the number set. The string may appear anywhere within a line and the line containing string is not considered part of the set. Matching for the string is case sensitive.
-numberLines NUMBER | -n NUMBER
The number of lines that are in the number set.
-csvFormat | -c
Indicates that output should be in comma separated values (csv) format, suitable for input into a spreadsheet. See figure 4, figure 5, figure 9, and figure 14.
-deltasOnly | -d
Indicates that output should include only the delta line. See figure 3 and figure 8.
-hex | -h
Indicates that all numbers will be treated as hexadecimal. See figure 15
-separator | -s
Indicates that the lines containing the unique string should be added to the block of separator lines that separate the data sets when printed. This is useful if the lines contains a date/time stamp or something else that uniquely identifies the data set. The format is line at the start of set 2 dash line at the start of set 1. See figure 17.
-outputFile PATH | -o PATH
The path of the output file. If the file already exists it is overwritten. See figure 5.
-loose | -l
Indicates that labels to not have to match exactly. See figure 12 and figure 13.
Notes:
1. Arguments can be in any order.
2. If no output path is indicated output will go to standard output.
Tested with
MS Windows: ActiveState Perl build 1004 [287188] build Sept 3, 2008, version v5.10.0 built for MSWin32-x86-multi-thread.
Linux/Ubuntu 10.04 LTS - the Lucid Lynx released April 2010: Perl version v5.10.1 (*) build for i486-linux-gnu-thread-multi.
Stratus OpenVOS 17.0.2ah: Perl version v5.8.0 built for i686-vos.
Linux/Ubuntu 15.10 - perl5 (revision 5 version 20 subversion 2)
Examples:
This first example is taken from a MS Windows system. I used the netstat-statistics.vbs script to generate the statistics. To reduce the size of the example I have deleted everything but the "TCP Statistics for IPv4" portion of the command's output.
================== 2010-10-25 4:15:39 PM - CORP\PARADISEVALLEY================== TCP Statistics for IPv4 Active Opens = 4583 Passive Opens = 50 Failed Connection Attempts = 1824 Reset Connections = 1194 Current Connections = 16 Segments Received = 174607 Segments Sent = 116376 Segments Retransmitted = 25737 ================== 2010-10-25 4:16:39 PM - CORP\PARADISEVALLEY================== TCP Statistics for IPv4 Active Opens = 4583 Passive Opens = 50 Failed Connection Attempts = 1824 Reset Connections = 1194 Current Connections = 15 Segments Received = 174620 Segments Sent = 116387 Segments Retransmitted = 25740 ================== 2010-10-25 4:17:40 PM - CORP\PARADISEVALLEY================== TCP Statistics for IPv4 Active Opens = 4586 Passive Opens = 50 Failed Connection Attempts = 1824 Reset Connections = 1196 Current Connections = 15 Segments Received = 174657 Segments Sent = 116422 Segments Retransmitted = 25749 ================== 2010-10-25 4:18:40 PM - CORP\PARADISEVALLEY================== TCP Statistics for IPv4 Active Opens = 4589 Passive Opens = 50 Failed Connection Attempts = 1824 Reset Connections = 1197 Current Connections = 17 Segments Received = 174733 Segments Sent = 116485 Segments Retransmitted = 25751 ================== 2010-10-25 4:19:40 PM - CORP\PARADISEVALLEY================== TCP Statistics for IPv4 Active Opens = 4591 Passive Opens = 50 Failed Connection Attempts = 1824 Reset Connections = 1198 Current Connections = 18 Segments Received = 175094 Segments Sent = 116712 Segments Retransmitted = 25757 ================== 2010-10-25 4:20:41 PM - CORP\PARADISEVALLEY================== TCP Statistics for IPv4 Active Opens = 4595 Passive Opens = 50 Failed Connection Attempts = 1824 Reset Connections = 1202 Current Connections = 18 Segments Received = 175248 Segments Sent = 116821 Segments Retransmitted = 25768 |
Again, in the interest of keeping these examples small I will only look at the segments sent, received and retransmitted. To do this I start the data set following the line containing the word "Current" and go for only 3 lines. The following figure shows the comparison output format. Each set of output includes the line from the current set of data, the previous set and the delta between any numbers of the line. Each number is right justified in a 15 character field, this allows large numbers and small differences to line up nicely but does produce a jagged display when the numbers are to the left of the labels.
Adding the -d (deltasOnly) suppresses the lines from the data sets and leaves only the delta lines.
Finally the -c (cvsFormat) generates something that you pull into a spreadsheet.
Typically you would also use the -o argument to send the output to a file. Note that the line repeating the command line with all arguments is not part of the output file.
These next examples are from the Ubuntu system. The output was generated by repeated execution of the "netstat -s" command. Again, in the interest of saving space I deleted everything but the "Ip" section of output.
Since the Ubuntu format has the numbers to the left of the labels everything aligns nicely. Also this shows an example of a line "Ip:" which does not include any numbers. These "labels only" lines appear only once in the comparison and deltas only output formats and not at all in the CSV format.
Finally, these last examples are from the Stratus OpenVOS system. The raw data are the first few lines of output from the stcp_meters analyze_system request.
As you can see from the raw data each line of the data set includes 2 numbers one of which is not an integer.
By default corresponding labels between data sets must match exactly. This helps catch instances when for some reason the data set changes. However, in some instances, for example a time label, the different values do not indicate a problem. The -loose argument can be used to override the default action which is to report the discrepancy and stop. Note that "***" on the "Metering time" line. this is an indication that the label does not match the label in the previous data set.
This example shows how the header row for the two columns are constructed when the CSV format is used. The first two lines have the format "label number label number label". For these lines the label to the left of the number is used and the final label is ignored. The second two lines have the format "label number number label". For these two lines the label to the left of the first number and to the right of the second number are used. In the event that there are more numbers then labels on the line a underscore character is used as a place holder.
This example shows the comparison output when dealing with a set of numbers displayed in hex. Again note that the delta values are always displayed in decimal.
This example shows the use of the -separator argument. Note that the string at the start of the data set a time stamp.
Figure 2 - comparison output format showing the input lines and delta values
L:\>perl delta.pl -i netstat-statistics.2010-10-25.txt -u Current -n 3
perl delta.pl -inputFile netstat-statistics.2010-10-25.txt -uniqueString Current
-numberLines 3
Segments Received = 174620
Segments Received = 174607
Segments Received = 13
Segments Sent = 116387
Segments Sent = 116376
Segments Sent = 11
Segments Retransmitted = 25740
Segments Retransmitted = 25737
Segments Retransmitted = 3
===============================================================
===============================================================
Segments Received = 174657
Segments Received = 174620
Segments Received = 37
Segments Sent = 116422
Segments Sent = 116387
Segments Sent = 35
Segments Retransmitted = 25749
Segments Retransmitted = 25740
Segments Retransmitted = 9
===============================================================
===============================================================
Segments Received = 174733
Segments Received = 174657
Segments Received = 76
Segments Sent = 116485
Segments Sent = 116422
Segments Sent = 63
Segments Retransmitted = 25751
Segments Retransmitted = 25749
Segments Retransmitted = 2
===============================================================
===============================================================
Segments Received = 175094
Segments Received = 174733
Segments Received = 361
Segments Sent = 116712
Segments Sent = 116485
Segments Sent = 227
Segments Retransmitted = 25757
Segments Retransmitted = 25751
Segments Retransmitted = 6
===============================================================
===============================================================
Segments Received = 175248
Segments Received = 175094
Segments Received = 154
Segments Sent = 116821
Segments Sent = 116712
Segments Sent = 109
Segments Retransmitted = 25768
Segments Retransmitted = 25757
Segments Retransmitted = 11
===============================================================
===============================================================
L:\>
Figure 3 - Deltas Only output format
L:\>perl delta.pl -i netstat-statistics.2010-10-25.txt -u Current -n 3 -d
perl delta.pl -inputFile netstat-statistics.2010-10-25.txt -uniqueString Current
-numberLines 3 -deltasOnly
Segments Received = 13
Segments Sent = 11
Segments Retransmitted = 3
===============================================================
===============================================================
Segments Received = 37
Segments Sent = 35
Segments Retransmitted = 9
===============================================================
===============================================================
Segments Received = 76
Segments Sent = 63
Segments Retransmitted = 2
===============================================================
===============================================================
Segments Received = 361
Segments Sent = 227
Segments Retransmitted = 6
===============================================================
===============================================================
Segments Received = 154
Segments Sent = 109
Segments Retransmitted = 11
===============================================================
===============================================================
L:\>
Figure 4 - CSV (comma separated values) output format
L:\>perl delta.pl -i netstat-statistics.2010-10-25.txt -u Current -n 3 -c
perl delta.pl -inputFile netstat-statistics.2010-10-25.txt -uniqueString Current
-numberLines 3 -csvFormat
Segments Received = , Segments Sent = , Segments Retransmitted = ,
13, 11, 3,
37, 35, 9,
76, 63, 2,
361, 227, 6,
154, 109, 11,
L:\>
Figure 5 - Creating a CSV formated file
L:\>perl delta.pl -i netstat-statistics.2010-10-25.txt -u Current -n 3 -c -o sta
ts.csv
perl delta.pl -inputFile netstat-statistics.2010-10-25.txt -uniqueString Current
-numberLines 3 -csvFormat -outputFile stats.csv
L:\>type stats.csv
Segments Received = , Segments Sent = , Segments Retransmitted = ,
13, 11, 3,
37, 35, 9,
76, 63, 2,
361, 227, 6,
154, 109, 11,
L:\>
Figure 6 - Ubuntu Linux raw data
netstat -s
Ip:
75065 total packets received
2089 with invalid addresses
0 forwarded
0 incoming packets discarded
72973 incoming packets delivered
4239 requests sent out
64 dropped because of missing route
netstat -s
Ip:
75273 total packets received
2101 with invalid addresses
0 forwarded
0 incoming packets discarded
73169 incoming packets delivered
4258 requests sent out
64 dropped because of missing route
netstat -s
Ip:
75497 total packets received
2113 with invalid addresses
0 forwarded
0 incoming packets discarded
73381 incoming packets delivered
4259 requests sent out
64 dropped because of missing route
netstat -s
Ip:
75699 total packets received
2125 with invalid addresses
0 forwarded
0 incoming packets discarded
73571 incoming packets delivered
4265 requests sent out
64 dropped because of missing route
netstat -s
Ip:
75914 total packets received
2137 with invalid addresses
0 forwarded
0 incoming packets discarded
73774 incoming packets delivered
4265 requests sent out
64 dropped because of missing route
netstat -s
Ip:
76150 total packets received
2149 with invalid addresses
0 forwarded
0 incoming packets discarded
73998 incoming packets delivered
4269 requests sent out
64 dropped because of missing route
Figure 7 - comparison output format
noah@noah-desktop:~/Desktop/delta$ perl delta.pl -i 10-10-25 -u netstat -n 5
perl delta.pl -inputFile 10-10-25 -uniqueString netstat -numberLines 5
Ip:
75273 total packets received
75065 total packets received
208 total packets received
2101 with invalid addresses
2089 with invalid addresses
12 with invalid addresses
0 forwarded
0 forwarded
0 forwarded
0 incoming packets discarded
0 incoming packets discarded
0 incoming packets discarded
===============================================================
===============================================================
Ip:
75497 total packets received
75273 total packets received
224 total packets received
2113 with invalid addresses
2101 with invalid addresses
12 with invalid addresses
0 forwarded
0 forwarded
0 forwarded
0 incoming packets discarded
0 incoming packets discarded
0 incoming packets discarded
===============================================================
===============================================================
Ip:
75699 total packets received
75497 total packets received
202 total packets received
2125 with invalid addresses
2113 with invalid addresses
12 with invalid addresses
0 forwarded
0 forwarded
0 forwarded
0 incoming packets discarded
0 incoming packets discarded
0 incoming packets discarded
===============================================================
===============================================================
Ip:
75914 total packets received
75699 total packets received
215 total packets received
2137 with invalid addresses
2125 with invalid addresses
12 with invalid addresses
0 forwarded
0 forwarded
0 forwarded
0 incoming packets discarded
0 incoming packets discarded
0 incoming packets discarded
===============================================================
===============================================================
Ip:
76150 total packets received
75914 total packets received
236 total packets received
2149 with invalid addresses
2137 with invalid addresses
12 with invalid addresses
0 forwarded
0 forwarded
0 forwarded
0 incoming packets discarded
0 incoming packets discarded
0 incoming packets discarded
===============================================================
===============================================================
Figure 8 - Deltas Only output format
noah@noah-desktop:~/Desktop/delta$ perl delta.pl -i 10-10-25 -u netstat -n 5 -d
perl delta.pl -inputFile 10-10-25 -uniqueString netstat -numberLines 5 -deltasO
nly
Ip:
208 total packets received
12 with invalid addresses
0 forwarded
0 incoming packets discarded
===============================================================
===============================================================
Ip:
224 total packets received
12 with invalid addresses
0 forwarded
0 incoming packets discarded
===============================================================
===============================================================
Ip:
202 total packets received
12 with invalid addresses
0 forwarded
0 incoming packets discarded
===============================================================
===============================================================
Ip:
215 total packets received
12 with invalid addresses
0 forwarded
0 incoming packets discarded
===============================================================
===============================================================
Ip:
236 total packets received
12 with invalid addresses
0 forwarded
0 incoming packets discarded
===============================================================
===============================================================
Figure 9 - CSV output format
noah@noah-desktop:~/Desktop/delta$ perl delta.pl -i 10-10-25 -u netstat -n 5 -c
perl delta.pl -inputFile 10-10-25 -uniqueString netstat -numberLines 5 -csvForm
at
total packets received , with invalid addresses , forwarded , incoming packets d
iscarded ,
208, 12, 0, 0,
224, 12, 0, 0,
202, 12, 0, 0,
215, 12, 0, 0,
236, 12, 0, 0,
Figure 10 - Stratus VOS raw data
as: stcp_meters -all -long
stcp_meters %phx_vos#m16 1008:29:49 10-09-22 13:22:30
STCP Version 4 loaded 10-08-11 12:52:41
Metering time: 1008:29:49
segments received 3488882128 (avg 960.96/sec)
total bytes 4385455957358 (avg 1207918.59/sec)
using fast mode 4360099101066 (99.4% of total bytes)
flow controlled when received 1665540 (0% of segments rcvd)
data pending when received 15672580 (0.4% of segments rcvd)
received out of order 14204065 (0.4% of segments rcvd)
as: stcp_meters -all -long
stcp_meters %phx_vos#m16 1008:44:49 10-09-22 13:37:30
STCP Version 4 loaded 10-08-11 12:52:41
Metering time: 1008:44:49
segments received 3488882934 (avg 960.73/sec)
total bytes 4385455998608 (avg 1207619.24/sec)
using fast mode 4360099142316 (99.4% of total bytes)
flow controlled when received 1665540 (0% of segments rcvd)
data pending when received 15672580 (0.4% of segments rcvd)
received out of order 14204065 (0.4% of segments rcvd)
as: stcp_meters -all -long
stcp_meters %phx_vos#m16 1008:59:49 10-09-22 13:52:30
STCP Version 4 loaded 10-08-11 12:52:41
Metering time: 1008:59:49
segments received 3488883774 (avg 960.49/sec)
total bytes 4385456041150 (avg 1207320.04/sec)
using fast mode 4360099184858 (99.4% of total bytes)
flow controlled when received 1665540 (0% of segments rcvd)
data pending when received 15672580 (0.4% of segments rcvd)
received out of order 14204067 (0.4% of segments rcvd)
as: stcp_meters -all -long
stcp_meters %phx_vos#m16 1009:14:50 10-09-22 14:07:31
STCP Version 4 loaded 10-08-11 12:52:41
Metering time: 1009:14:50
segments received 3488884556 (avg 960.25/sec)
total bytes 4385456081488 (avg 1207020.65/sec)
using fast mode 4360099225196 (99.4% of total bytes)
flow controlled when received 1665540 (0% of segments rcvd)
data pending when received 15672580 (0.4% of segments rcvd)
received out of order 14204067 (0.4% of segments rcvd)
as: stcp_meters -all -long
stcp_meters %phx_vos#m16 1009:29:50 10-09-22 14:22:31
STCP Version 4 loaded 10-08-11 12:52:41
Metering time: 1009:29:50
segments received 3488885376 (avg 960.01/sec)
total bytes 4385456123270 (avg 1206721.75/sec)
using fast mode 4360099266978 (99.4% of total bytes)
flow controlled when received 1665540 (0% of segments rcvd)
data pending when received 15672580 (0.4% of segments rcvd)
received out of order 14204068 (0.4% of segments rcvd)
as: stcp_meters -all -long
stcp_meters %phx_vos#m16 1009:44:50 10-09-22 14:37:31
STCP Version 4 loaded 10-08-11 12:52:41
Metering time: 1009:44:50
segments received 3488886274 (avg 959.77/sec)
total bytes 4385456168016 (avg 1206422.99/sec)
using fast mode 4360099311724 (99.4% of total bytes)
flow controlled when received 1665540 (0% of segments rcvd)
data pending when received 15672580 (0.4% of segments rcvd)
received out of order 14204068 (0.4% of segments rcvd)
as: stcp_meters -all -long
Figure 11 - 2 columns of numbers and non-integer numbers
perl delta.pl -i meters -u Metering -n 5
perl delta.pl -inputFile meters -uniqueString Metering -numberLines 5
segments received 3488882934 avg 960.73 sec
segments received 3488882128 avg 960.96 sec
segments received 806 avg -0.23 sec
total bytes 4385455998608 avg 1207619.24 sec
total bytes 4385455957358 avg 1207918.59 sec
total bytes 41250 avg -299.35 sec
using fast mode 4360099142316 99.4 of total bytes
using fast mode 4360099101066 99.4 of total bytes
using fast mode 41250 0.0 of total bytes
flow controlled when received 1665540 0 of segments rcvd
+
flow controlled when received 1665540 0 of segments rcvd
+
flow controlled when received 0 0 of segments rcvd
+
===============================================================
===============================================================
segments received 3488883774 avg 960.49 sec
segments received 3488882934 avg 960.73 sec
segments received 840 avg -0.24 sec
total bytes 4385456041150 avg 1207320.04 sec
total bytes 4385455998608 avg 1207619.24 sec
total bytes 42542 avg -299.20 sec
using fast mode 4360099184858 99.4 of total bytes
using fast mode 4360099142316 99.4 of total bytes
using fast mode 42542 0.0 of total bytes
flow controlled when received 1665540 0 of segments rcvd
+
flow controlled when received 1665540 0 of segments rcvd
+
flow controlled when received 0 0 of segments rcvd
+
===============================================================
===============================================================
segments received 3488884556 avg 960.25 sec
segments received 3488883774 avg 960.49 sec
segments received 782 avg -0.24 sec
total bytes 4385456081488 avg 1207020.65 sec
total bytes 4385456041150 avg 1207320.04 sec
total bytes 40338 avg -299.39 sec
using fast mode 4360099225196 99.4 of total bytes
using fast mode 4360099184858 99.4 of total bytes
using fast mode 40338 0.0 of total bytes
flow controlled when received 1665540 0 of segments rcvd
+
flow controlled when received 1665540 0 of segments rcvd
+
flow controlled when received 0 0 of segments rcvd
+
===============================================================
===============================================================
segments received 3488885376 avg 960.01 sec
segments received 3488884556 avg 960.25 sec
segments received 820 avg -0.24 sec
total bytes 4385456123270 avg 1206721.75 sec
total bytes 4385456081488 avg 1207020.65 sec
total bytes 41782 avg -298.90 sec
using fast mode 4360099266978 99.4 of total bytes
using fast mode 4360099225196 99.4 of total bytes
using fast mode 41782 0.0 of total bytes
flow controlled when received 1665540 0 of segments rcvd
+
flow controlled when received 1665540 0 of segments rcvd
+
flow controlled when received 0 0 of segments rcvd
+
===============================================================
===============================================================
segments received 3488886274 avg 959.77 sec
segments received 3488885376 avg 960.01 sec
segments received 898 avg -0.24 sec
total bytes 4385456168016 avg 1206422.99 sec
total bytes 4385456123270 avg 1206721.75 sec
total bytes 44746 avg -298.76 sec
using fast mode 4360099311724 99.4 of total bytes
using fast mode 4360099266978 99.4 of total bytes
using fast mode 44746 0.0 of total bytes
flow controlled when received 1665540 0 of segments rcvd
+
flow controlled when received 1665540 0 of segments rcvd
+
flow controlled when received 0 0 of segments rcvd
+
===============================================================
===============================================================
ready 15:17:30
Figure 12 - Program termination because the labels do not match
perl delta.pl -i meters -u Version -n 6
perl delta.pl -inputFile meters -uniqueString Version -numberLines 6
[Metering time: 1008:44:49 ] and [Metering time: 1008:29:49 ] do not match at in
+dex 0
ready 15:35:20
Figure 13 - Using the -loose argument to ignore label differences
perl delta.pl -i meters -u Version -n 6 -l
perl delta.pl -inputFile meters -uniqueString Version -numberLines 6 -loose
****Metering time: 1008:44:49 ***
segments received 3488882934 avg 960.73 sec
segments received 3488882128 avg 960.96 sec
segments received 806 avg -0.23 sec
total bytes 4385455998608 avg 1207619.24 sec
total bytes 4385455957358 avg 1207918.59 sec
total bytes 41250 avg -299.35 sec
using fast mode 4360099142316 99.4 of total bytes
using fast mode 4360099101066 99.4 of total bytes
using fast mode 41250 0.0 of total bytes
flow controlled when received 1665540 0 of segments rcvd
+
flow controlled when received 1665540 0 of segments rcvd
+
flow controlled when received 0 0 of segments rcvd
+
===============================================================
===============================================================
****Metering time: 1008:59:49 ***
segments received 3488883774 avg 960.49 sec
segments received 3488882934 avg 960.73 sec
segments received 840 avg -0.24 sec
total bytes 4385456041150 avg 1207320.04 sec
total bytes 4385455998608 avg 1207619.24 sec
total bytes 42542 avg -299.20 sec
using fast mode 4360099184858 99.4 of total bytes
using fast mode 4360099142316 99.4 of total bytes
using fast mode 42542 0.0 of total bytes
flow controlled when received 1665540 0 of segments rcvd
+
flow controlled when received 1665540 0 of segments rcvd
+
flow controlled when received 0 0 of segments rcvd
+
===============================================================
===============================================================
****Metering time: 1009:14:50 ***
segments received 3488884556 avg 960.25 sec
segments received 3488883774 avg 960.49 sec
segments received 782 avg -0.24 sec
total bytes 4385456081488 avg 1207020.65 sec
total bytes 4385456041150 avg 1207320.04 sec
total bytes 40338 avg -299.39 sec
using fast mode 4360099225196 99.4 of total bytes
using fast mode 4360099184858 99.4 of total bytes
using fast mode 40338 0.0 of total bytes
flow controlled when received 1665540 0 of segments rcvd
+
flow controlled when received 1665540 0 of segments rcvd
+
flow controlled when received 0 0 of segments rcvd
+
===============================================================
===============================================================
****Metering time: 1009:29:50 ***
segments received 3488885376 avg 960.01 sec
segments received 3488884556 avg 960.25 sec
segments received 820 avg -0.24 sec
total bytes 4385456123270 avg 1206721.75 sec
total bytes 4385456081488 avg 1207020.65 sec
total bytes 41782 avg -298.90 sec
using fast mode 4360099266978 99.4 of total bytes
using fast mode 4360099225196 99.4 of total bytes
using fast mode 41782 0.0 of total bytes
flow controlled when received 1665540 0 of segments rcvd
+
flow controlled when received 1665540 0 of segments rcvd
+
flow controlled when received 0 0 of segments rcvd
+
===============================================================
===============================================================
****Metering time: 1009:44:50 ***
segments received 3488886274 avg 959.77 sec
segments received 3488885376 avg 960.01 sec
segments received 898 avg -0.24 sec
total bytes 4385456168016 avg 1206422.99 sec
total bytes 4385456123270 avg 1206721.75 sec
total bytes 44746 avg -298.76 sec
using fast mode 4360099311724 99.4 of total bytes
using fast mode 4360099266978 99.4 of total bytes
using fast mode 44746 0.0 of total bytes
flow controlled when received 1665540 0 of segments rcvd
+
flow controlled when received 1665540 0 of segments rcvd
+
flow controlled when received 0 0 of segments rcvd
+
===============================================================
===============================================================
ready 15:35:44
Figure 14 - CSV format derived from multi-number lines
perl delta.pl -i meters -u Metering -n 5 -c
perl delta.pl -inputFile meters -uniqueString Metering -numberLines 5 -csvForma
+t
segments received , avg , total bytes , avg , using fast mode , of total bytes ,
+ flow controlled when received , of segments rcvd ,
806, -0.23, 41250, -299.35, 41250, 0.0, 0, 0,
840, -0.24, 42542, -299.20, 42542, 0.0, 0, 0,
782, -0.24, 40338, -299.39, 40338, 0.0, 0, 0,
820, -0.24, 41782, -298.90, 41782, 0.0, 0, 0,
898, -0.24, 44746, -298.76, 44746, 0.0, 0, 0,
ready 18:01:08
Figure 15 - dealing with hexadecimal numbers
perl delta.pl -i mib_stats -u mib -n 6 -h
perl delta.pl -inputFile mib_stats -uniqueString mib -numberLines 6 -hex
version = 1
version = 1
version = 0
index = 00000011
index = 00000011
index = 0
Rx octets received = 581D1F2E
Rx octets received = 5812C4A5
Rx octets received = 678537
Rx unicast packets delivered = 4BA230D1
Rx unicast packets delivered = 4BA22E30
Rx unicast packets delivered = 673
Rx broadcasts multicast packets = 012688DF
Rx broadcasts multicast packets = 01266C07
Rx broadcasts multicast packets = 7384
===============================================================
===============================================================
version = 1
version = 1
version = 0
index = 00000011
index = 00000011
index = 0
Rx octets received = 58256D06
Rx octets received = 581D1F2E
Rx octets received = 544216
Rx unicast packets delivered = 4BA23374
Rx unicast packets delivered = 4BA230D1
Rx unicast packets delivered = 675
Rx broadcasts multicast packets = 0126A016
Rx broadcasts multicast packets = 012688DF
Rx broadcasts multicast packets = 5943
===============================================================
===============================================================
version = 1
version = 1
version = 0
index = 00000011
index = 00000011
index = 0
Rx octets received = 582BA692
Rx octets received = 58256D06
Rx octets received = 407948
Rx unicast packets delivered = 4BA23614
Rx unicast packets delivered = 4BA23374
Rx unicast packets delivered = 672
Rx broadcasts multicast packets = 0126B20D
Rx broadcasts multicast packets = 0126A016
Rx broadcasts multicast packets = 4599
===============================================================
===============================================================
version = 1
version = 1
version = 0
index = 00000011
index = 00000011
index = 0
Rx octets received = 583206E3
Rx octets received = 582BA692
Rx octets received = 417873
Rx unicast packets delivered = 4BA238B5
Rx unicast packets delivered = 4BA23614
Rx unicast packets delivered = 673
Rx broadcasts multicast packets = 0126C439
Rx broadcasts multicast packets = 0126B20D
Rx broadcasts multicast packets = 4652
===============================================================
===============================================================
version = 1
version = 1
version = 0
index = 00000011
index = 00000011
index = 0
Rx octets received = 5838189B
Rx octets received = 583206E3
Rx octets received = 397752
Rx unicast packets delivered = 4BA23B55
Rx unicast packets delivered = 4BA238B5
Rx unicast packets delivered = 672
Rx broadcasts multicast packets = 0126D5E3
Rx broadcasts multicast packets = 0126C439
Rx broadcasts multicast packets = 4522
===============================================================
===============================================================
ready 17:56:12
Figure 16 - Example data set with time stamp as part of the unique string
Sat Mar 26 06:41:52 MST 2016
Ip:
2508 total packets received
2 with invalid addresses
0 forwarded
0 incoming packets discarded
2506 incoming packets delivered
2125 requests sent out
Icmp:
--
Sat Mar 26 06:41:57 MST 2016
Ip:
2539 total packets received
2 with invalid addresses
0 forwarded
0 incoming packets discarded
2537 incoming packets delivered
2156 requests sent out
Icmp:
--
Sat Mar 26 06:42:02 MST 2016
Ip:
2541 total packets received
2 with invalid addresses
0 forwarded
0 incoming packets discarded
2539 incoming packets delivered
2158 requests sent out
Icmp:
--
Sat Mar 26 06:42:07 MST 2016
Ip:
4304 total packets received
2 with invalid addresses
0 forwarded
0 incoming packets discarded
4302 incoming packets delivered
5556 requests sent out
Icmp:
Figure 17 - Example use of the -separator argument
$ perl delta.pl -in ip_stats.txt -un "MST 2016" -nu 8 -d -s
perl delta.pl -inputFile ip_stats.txt -uniqueString MST 2016 -numberLines 8
-deltasOnly -separator
Ip:
31 total packets received
0 with invalid addresses
0 forwarded
0 incoming packets discarded
31 incoming packets delivered
31 requests sent out
Icmp:
===============================================================
Sat Mar 26 06:41:57 MST 2016 - Sat Mar 26 06:41:52 MST 2016
===============================================================
Ip:
2 total packets received
0 with invalid addresses
0 forwarded
0 incoming packets discarded
2 incoming packets delivered
2 requests sent out
Icmp:
===============================================================
Sat Mar 26 06:42:02 MST 2016 - Sat Mar 26 06:41:57 MST 2016
===============================================================
Ip:
1763 total packets received
0 with invalid addresses
0 forwarded
0 incoming packets discarded
1763 incoming packets delivered
3398 requests sent out
Icmp:
===============================================================
Sat Mar 26 06:42:07 MST 2016 - Sat Mar 26 06:42:02 MST 2016
===============================================================
$
delta.pl
# delta.pl begins here
#
# Version 1.00 10-11-07
# Version 1.10 10-11-26 Added disclaimer
# Version 1.20 16-03-26 Added "include separator line" argument
#
# noah@noahdavids.org
#
# See http://noahdavids.org/self_published/delta.html for documentation
#
# This software is provided on an "AS IS" basis, WITHOUT ANY WARRANTY OR ANY
# SUPPORT OF ANY KIND. The AUTHOR SPECIFICALLY DISCLAIMS ANY IMPLIED WARRANTIES
# OF MERCHANTABILITY OR FITNESS FOR ANY PARTICULAR PURPOSE. This disclaimer
# applies, despite any verbal representations of any kind provided by the
# author or anyone else.
#
use Getopt::Long;
use strict;
my ($result);
my ($uniqueString, $uniqueStringLine1, $ uniqueStringLine2, $uniqueStringLineLen);
my ($numberLines, $csvFormat, $deltasOnly, $hexNumbers,
$looseMatch, $includeSeparator, $inputFile, $outputFile);
my ($message);
my ($INFILE, $OUTFILE);
my (@set, @set1, @maxIndex, @numberCount, $headerCount, $a, $i, $j, $k, $mi,
$nc, $numCount);
my ($temp, $temp1, $temp2);
my ($spaces);
my ($decimalDigits, $formatString);
my ($x);
my ($decmatch, $hexmatch, $match);
$decmatch = "^\[+-\]\?\[0-9\]\*\\\.\?\[0-9\]\*\$";
$hexmatch = "^\[+-\]\?\[a-fA-F0-9\]\*\\\.\?\[a-fA-F0-9\]\*\$";
sub printIt
{
# If the value to be printed is a number and we are not doing CSV format
# justify the number to 15 places. If it is not a number just print it, if
# its a number but we are doing CSV format just print it. I figure most
# number fields will be smaller than 15 places and this will right justify
# large numbers and possibly small differences. It works fine when all the
# numbers come first. When numbers come last or in the middle you can end up
# with a jagged column of numbers when the labels are different lengths. You
# can't have everything.
if ($_[0] =~ /$match/)
{
if (!defined ($csvFormat))
{
$temp = 15 - length ($_[0]);
if ($temp > 0) { print $OUTFILE substr ($spaces, 1, $temp) ; }
}
print $OUTFILE $_[0];
}
else { print $OUTFILE $_[0]; }
}
sub parseLine
{
my (@aLine);
my ($a, $i, $k, $lastWasLabel);
$_ = <$INFILE>;
# Split the line up into tokens. Delimiters are any amount of white
# space (\s+) also the characters / (as in N/sec), % as in (N.M%) and
# ( and ) as in (N). Unfortunately, this elimiates the single characters
# from the labels but I figure it is better to have the delta calculated
# rather than treat the field as a label and just display it.
@aLine = split(/\s+|\/|\%|\(|\)/);
$a = @aLine;
$i = -1;
$set[0] = "";
$lastWasLabel = 0;
$nc = 0;
for ($k = 0; $k < $a; $k++)
{
if (length ($aLine[$k]) > 0)
{
if (!($aLine[$k] =~ /$match/))
{
# If we are currently looking at a label (not a number) and
# if the last thing we looked at was not a label increment
# the count and copy the label. If the last thing we looked
# at was a label append the current label onto the last label.
if (!$lastWasLabel) { $i++ ; $set[$i] = ""; }
$set [$i] = $set [$i] . $aLine[$k] . " ";
$lastWasLabel = 1;
}
else
{
# We are looking at a number, increment the count, copy
# the number and increment the number count (nc)
$i++;
$set [$i] = $aLine[$k];
$lastWasLabel = 0;
$nc++;
}
}
} # for ($k = 0; $k < $a; $k++)
# Increment the maximum index (mi) for this row.
$mi = $i + 1;
} # end of parseLine
$spaces = " ";
$result = GetOptions ('uniqueString=s' => \$uniqueString,
'numberLines=s' => \$numberLines,
'csvFormat' => \$csvFormat,
'deltasOnly' => \$deltasOnly,
'hex' => \$hexNumbers,
'loose' => \$looseMatch,
'separator' => \$includeSeparator,
'inputFile=s' => \$inputFile,
'outputFile=s' => \$outputFile);
# There are 3 required arguments, the input file, the unique string and the
# number ofd lines in the data set. If those arguments aren't supplied print
# a usage message and exit
if (!(($result == 1) && (defined($inputFile) && defined($uniqueString) &&
defined($numberLines))))
{
print "\n\nUsage:\n";
print "\tperl delta.pl -inputFile PATH -uniqueString STRING -numberLines" .
" NUMBER [-csvFormat] [-deltasOnly] [-hex]" .
" [separator] [-outputFile PATH] [-loose]\n\n";
exit;
}
# Build a message with all the optional arguments tha were provided. Print
# out the command line and 3 required arguments along with the optional
# argument message. I like to see what arguments the command was called with.
$message = " ";
if (defined($deltasOnly)) { $message = $message . " -deltasOnly"; }
if (defined($csvFormat)) { $message = $message . " -csvFormat"; }
if (defined($hexNumbers)) { $message = $message . " -hex"; }
if (defined($looseMatch)) { $message = $message . " -loose"; }
if (defined($includeSeparator)) { $message = $message . " -separator"; }
if (defined($outputFile)) { $message = $message . " -outputFile " .
$outputFile; }
$message = $message . "\n\n";
print "perl delta.pl -inputFile " . $inputFile . " -uniqueString " .
$uniqueString . " -numberLines " . $numberLines . $message;
# Set the "number" match string to match either decimal or hexidecimal numbers
# based on whether the -hex argument was provided or not.
if (defined ($hexNumbers)) { $match = $hexmatch; }
else { $match = $decmatch; }
# Open the input file and if provided the output file or STDOUT if no output
# file path was provided.
open ($INFILE, $inputFile);
if (!defined($outputFile)) {open ($OUTFILE, ">&STDOUT");}
else {open ($OUTFILE, ">".$outputFile) ||
die "Can't open outfile " . $outputFile;}
# Search for the first instance of the unique string.
while ($_ = <$INFILE>) { if (/$uniqueString/) { last } }
if (length ($_) == 0) { exit (0); }
# Save line that unique string is on (trim off new line)
$uniqueStringLine1 = substr($_, 0, -1);
# Now parse the indicated number of lines building an array of fields for each
# line. Also arrays holding how many fields are in each line and how many
# fields are numbers.
for ($i = 0; $i < $numberLines; $i++)
{
parseLine ();
push @set1, [@set];
push @maxIndex, $mi;
push @numberCount, $nc;
}
# If writing a CSV formated file for import into a spreadsheet write the first
# row containing the column headers.
if (defined($csvFormat))
{
for ($i = 0; $i < $numberLines; $i++)
{
$a = $maxIndex [$i];
$headerCount = 0;
$numCount = 0;
# If a row does not contain any numbers skip it.
if ($numberCount [$i] > 0)
{
for ($j = 0; $j < $a; $j++)
{
# If the field is not a number we probably want to print it
# for a column header.
if (!($set1[$i][$j] =~ /$match/))
{
# Just incase the label has any commas translate them to
# semi-colons so that it will not screw up the import into
# a spreadsheet using commas to delimit the fields. Keep
# track of how many column headers we print.
$temp = $set1 [$i][$j];
$temp =~ tr/,/;/;
print $OUTFILE $temp . ", ";
$headerCount++;
}
# If the current field is not a label, its a number. If the
# next field is also a number and the count of headers printed
# is less than the count of numbers seen print a place holder
# header (and increment the header count).
elsif ($j+1 < $a)
{
$numCount++;
if ($set1[$i][$j+1] =~ /$match/)
{
if ($headerCount < $numCount)
{
print $OUTFILE "_, ";
$headerCount++;
}
}
}
# Once we have printed as many headers as there are numbers in
# that row stop.
if ($headerCount == $numberCount[$i]) { last; }
}
# Make sure that we have as many column headers as there are
# numbers in the row.
for ($j = $headerCount; $j < $numberCount[$i]; $j++)
{ print $OUTFILE "_, "; }
} # if ($numberCount [$i] > 0)
} # for ($i = 0; $i < $numberLines; $i++)
print $OUTFILE "\n";
# One last thing. If doing CSV format we do not need previous and current
# lines printed so set the deltasOnly flag.
$deltasOnly = 1;
} # if (defined($csvFormat))
do {
# Search for the next instance of the uniqueu string.
while ($_ = <$INFILE>) { if (/$uniqueString/) { last } }
if (length ($_) == 0) { exit (0); }
# Save line that unique string is on (trim off new line)
$uniqueStringLine2 = substr($_, 0, -1);
for ($i = 0; $i < $numberLines; $i++)
{
# Now parse the indicated number of lines.
parseLine ();
$a = $maxIndex [$i];
# If we are NOT doing just the delta lines print out the line we just
# read followed by the previous line. For number fields this should
# put the the larger number of top.
if (!defined($deltasOnly))
{
if ($numberCount [$i] > 0)
{
print $OUTFILE "\n";
for ($j = 0; $j < $a; $j++) { printIt ($set[$j]);
print $OUTFILE " "; }
print $OUTFILE "\n";
for ($j = 0; $j < $a; $j++) { printIt ($set1[$i][$j]);
print $OUTFILE " "; }
print $OUTFILE "\n";
}
}
# For each field in the line
for ($j = 0; $j < $a; $j++)
{
# If the current field is a number
if ($set [$j] =~ /$match/)
{
# And the corresponding field from the previous line is a number
# we need to calcuate the difference.
if ($set1[$i][$j] =~ /$match/)
{
# If we are NOT doing hex just subtract the numbers. If we are
# doing hex we convert from hex to decimal before subtracting.
# The convertion of the numbers to start with is needed because
# if the numbers are in hex and the values look like they are
# decimal, i.e. 200 and 90 perl will treat them as decimal not
# hex. NOTE that the result is always in decimal.
if (!defined($hexNumbers))
{
# You can't really just subtract the numbers. The
# problem is that if the numbers contain a decimal
# point Perl will give you all sorts of insignificant
# numbers to the right of the decimal point so you
# need to get rid of them. So the first thing to do
# is to see there are any decimal digits, if not just
# subtract and print.
if ($set [$j] - int ($set [$j]) == 0)
{ printIt ($set [$j] - $set1[$i][$j]); }
else
{
# First count how many decimal digits there are
# in the current number, then create a format string
# to present that number of decimal digits and print
# the number generated by that string.
$decimalDigits =
length ($set [$j]) - index ($set [$j], ".") - 1;
$formatString = sprintf ("%%.%df", $decimalDigits);
printIt (sprintf
($formatString, ($set [$j] - $set1[$i][$j])));
}
} # if (!defined($hexNumbers))
# Its hex, just convert, subtract and print
else { printIt (hex ($set [$j]) - hex ($set1[$i][$j])); }
# If doing CSV format we need a comma after the number, if not
# just print a space.
if (defined($csvFormat)) { print $OUTFILE ", "; }
else { print $OUTFILE " "; }
}
else
# If the current field in the current line is a number but the
# corresponding field in the previous set of lines is not a
# number report an error and exit.
{
print $OUTFILE "\n\nAt index " . $j . " [" . $set [$j] .
"] and [" . $set1[$i][$j] . "] do not match \n";
exit (1);
}
} # if ($set [$j] =~ /$match/)
else
# We are here because the current field in the current line is
# not a number. If the corresponding field in the previous line
# has the same value and we are not doing CSV format just print
# it. If the fields are not equal and the previous field is a
# number exit with an error. If both fields are just labels and
# looseMatch flag is not defined print an error and exit. If
# the looseMatch flag is defined and we are not doing CSV format,
# print it with some extra asterisks. Note that all the error
# messages are slightly different so we can tell which condition
# we hit.
{
if ($set [$j] eq $set1[$i][$j])
{
if (!defined($csvFormat))
{ print $OUTFILE $set[$j] . " "; }
}
else
{
if ($set1[$i][$j] =~ /$match/)
{
print $OUTFILE "\n\n<" . $set [$j] . "> and <" .
$set1[$i][$j] . "> do not match at index " . $j .
"\n";
exit (1);
}
elsif (!(defined ($looseMatch)))
{
print $OUTFILE "\n\n[" . $set [$j] . "] and [" .
$set1[$i][$j] . "] do not match at index " . $j .
"\n";
exit (1);
}
if (!defined($csvFormat)) { print $OUTFILE "****" .
$set[$j] . "*** "; }
}
}
# Copy the current value into the "old" array
$set1[$i][$j] = $set [$j];
} # for ($j = 0; $j < $a; $j++)
# If doing CSV format we need to output a new line at this point.
if (!defined($csvFormat)) { print $OUTFILE "\n"; }
} # for ($i = 0; $i < $numberLines; $i++)
# We are at the end of a data set, if we are not doing CSV format print a set
# of separator lines, if we are doing CSV format just print a new line
if (!defined($csvFormat))
{ $uniqueStringLineLen = length ($uniqueStringLine1) + length ($uniqueStringLine2);
print $OUTFILE
"\n===============================================================\n";
if (defined ($includeSeparator))
{
for ($x = 0; $x < ((60 - $uniqueStringLineLen) / 2); $x++)
{ print $OUTFILE " "; }
print $OUTFILE $uniqueStringLine2 . " - " . $uniqueStringLine1 ."\n";
}
print $OUTFILE
"===============================================================\n\n";
}
else { print $OUTFILE "\n"; }
$uniqueStringLine1 = $uniqueStringLine2;
} while (1); # do {
#
# delta.pl ends here
This page was last modified on 16-03-26
Send comments and suggestions
to noah@noahdavids.org