A Perl script to match lines from files
The following match.pl perl script can be used to display selective lines from a file or files or from the output of a command. You can provide multiple match strings and lines matching any of the strings will be displayed. Since the strings are regular expressions you can construct an expression that will require a line to have 2 or more strings in it in order to be a match. But you are saying to yourself - I can just use grep. But grep cannot also display all lines that contain ABC but not also XYZ.
Other things you can do, make the matches caseless (default is case sensitive), display line numbers, display the matching line and a some number of following lines and display a line of dots for skipped lines.
The script will work with VOS provided that the gnu_library is installed, will work on ftLinux which has Perl installed by default and will work on ftServer provided some version of Perl is installed (I tested with ActivePerl (www.activestate.com).
Usage
perl match.pl [-caseless] [-numbers] [-dots]
[-lines NUMBER] [-output PATH]
-match STRING [-match STRING]* [-except STRING]*
[[-file PATH]* | [-command STRING]]
-caseless
Indicates that the match should be caseless, the default is a case sensitive match. See example 9.
-numbers
Indicates that line numbers should be printed, the default is not to print line numbers. See example 6 and example 7.
-dots
Indicates that a line of dots should be printed between non-continuous printed lines. See example 7 and example 8.
-lines NUMBER
Number of lines to display. the default is 1. See example 8
-output PATH
Path to an output file. If not given then STDOUT is assumed and output goes to the teminal window. See example 13.
-match STRING
This is the match string. It can be a constant "foo" (example 1)or a regular expression "foo.*bar" (example 3). There must be at least 1 match string but there can be more (example 2 and example 6). There is probably a limit by Perl is pretty flexable. I've tested it with 10. The script may print any line that matches any of the strings (see except).
-except STRING
Any line that matches one of the match strings that also matches one of the except strings is NOT printed. See example 5.
-file PATH
Path to a file or files. Star names can be used. VOS ">" and "<" characters are translated into Unix style "/" and "../" characters. The Unix style is used when printing the file name. There must be at least 1 -file given but there can be more.
-command STRING
String represents a command to be executed (example 14). If the string has any spaces you must enclose it with quotes. If you wish to execute moe than 1 command you must place the commands in a script of VOS command macro.For VOS users you must also execute the commandmacro with the sh command (example 15)>.
Notes:
1. With the exception of the match, command and caseless arguments any substring of the argument string can be used, that is -e can be used for -except, -f for -file, etc. For command use must use at least co and for caseless at least ca, for match you must use -ma. The reason for command and caseless is that -c would be ambiguous. The reason for match is that -m is also ambiguous, there is an (undocumented) -module argument, this argument also fills in the match array I added it because so many VOS users have a "-m" abbreviation that maps to -module.
2. Arguments can be in any order.
Some examples:
The examples are from VOS but with the exception of using "/" instead of ">" to specify file names the same syntax will work. Note that program uses the "/" character when displaying the file paths.
Example 1 - your basic match
This is matching the string "case" in match.pl
perl match.pl -match case -file match.pl
********************************* match.pl *********************************
$minLines, $caseless, $showLineNumbers, $dots);
$caseless = 0;
'caseless' => \$caseless,
print "perl match.pl [-caseless] [-numbers] [-dots] \n" .
print "perl match.pl [-caseless] [-numbers] [-dots] \n" .
if ($caseless)
if ($caseless)
ready 10:20:39
|
Example 2 - matching multiple strings
This is matching the string "case" or "substr" in match.pl. Notice also that one -match argument preceeds the -file and the other follows it.
perl match.pl -match case -file match.pl -match substr
********************************* match.pl *********************************
$minLines, $caseless, $showLineNumbers, $dots);
$caseless = 0;
'caseless' => \$caseless,
print "perl match.pl [-caseless] [-numbers] [-dots] \n" .
print "perl match.pl [-caseless] [-numbers] [-dots] \n" .
print OUT substr ($stars, 1, (75 - length ($files[$b]))/2) . " " .
substr ($stars, 1, (75 - length ($files[$b]))/2) . "\n";
if ($caseless)
if ($caseless)
ready 10:21:39
|
Example 3 - matching lines containg two strings
This is matching the string "if" and "case". It does required that the "case" follow the "if". You could also match on "case.*if" to make it order insensitive. I admit this doesn't case well beyond 2 order insensitive strings - but how often do you want to do that.
perl match.pl -match if.*case -file match.pl
********************************* match.pl *********************************
if ($caseless)
if ($caseless)
ready 10:22:06
|
Example 4 - excluding lines from the match
This is matching the string "case" but NOT "if".
perl match.pl -match case -file match.pl -except if
********************************* match.pl *********************************
$minLines, $caseless, $showLineNumbers, $dots);
$caseless = 0;
'caseless' => \$caseless,
print "perl match.pl [-caseless] [-numbers] [-dots] \n" .
print "perl match.pl [-caseless] [-numbers] [-dots] \n" .
ready 10:22:36
|
Example 5 - all lines in a file EXCEPT some, this is grep -v
The "." matches all lines and the -except excludes lines with "if"
perl match.pl -match . -file match.pl -except if
********************************* match.pl *********************************
# match.pl begins here
#
# match.pl
# version 1.0 08-04-18
# version 1.1 08-05-20 added output switch to send output to a file instead of
# terminal screen
# version 1.2 09-10-08 added commandString argument to filter on command
# output instead of a file
# version 1.3 10-11-26 added disclaimer
# Noah.Davids@stratus.com
#
# The latest version of this macro and documentation can be found at
# http://noahdavids.org/self_published/match.html
#
# This software is provided on an "AS IS" basis, WITHOUT ANY WARRANTY OR ANY
# SUPPORT OF ANY KIND. The AUTHOR SPECIFICALLY DISCLAIMS ANY IMPLIED WARRANTIES
# OF MERCHANTABILITY OR FITNESS FOR ANY PARTICULAR PURPOSE. This disclaimer
# applies, despite any verbal representations of any kind provided by the
# author or anyone else.
#
use strict;
use warnings;
use Getopt::Long;
use Digest::MD5;
my (@match, @except, @inputFile, @outputFile, @commandString,
$minLines, $caseless, $showLineNumbers, $dots);
my (@files);
my ($INFILE);
my ($result, $a, $b, $i, $j, $print, $lineNumber, $lastLinePrinted, $reset);
--PAUSE--
|
Example 6 - line numbers
This is matching on case or stars but excluding if. It also adds line numbers. I also got tired of using the full argument name so everything is now 1 or 2 characters. If you have a "-m" abbreviations that maps to -module you can get away with using -m instead of -ma.
perl match.pl -ma case -f match.pl -ma stars -e if -n ********************************* match.pl ********************************* 27 $minLines, $caseless, $showLineNumbers, $dots); 31 my ($stars); 35 $caseless = 0; 38 $stars = "**************************************************"; 42 'caseless' => \$caseless, 56 print "perl match.pl [-caseless] [-numbers] [-dots] \n" . 66 print "perl match.pl [-caseless] [-numbers] [-dots] \n" . 96 print OUT substr ($stars, 1, (75 - length ($files[$b]))/2) . " " . 98 substr ($stars, 1, (75 - length ($files[$b]))/2) . "\n"; ready 10:25:24 |
Example 7 - dots
This is the same match as example 6 but I have added the dotted lines.
perl match.pl -ma case -f match.pl -ma stars -e if -n -d
********************************* match.pl *********************************
. . .
27 $minLines, $caseless, $showLineNumbers, $dots);
. . .
31 my ($stars);
. . .
35 $caseless = 0;
. . .
38 $stars = "**************************************************";
. . .
42 'caseless' => \$caseless,
. . .
56 print "perl match.pl [-caseless] [-numbers] [-dots] \n" .
. . .
66 print "perl match.pl [-caseless] [-numbers] [-dots] \n" .
. . .
96 print OUT substr ($stars, 1, (75 - length ($files[$b]))/2) . " " .
. . .
98 substr ($stars, 1, (75 - length ($files[$b]))/2) . "\n";
. . .
ready 10:28:41
|
Example 8 - minimum number of lines
This is the same as example 7 but with a minimum line count of 2
perl match.pl -ma case -f match.pl -ma stars -e if -n -d -l 2
********************************* match.pl *********************************
. . .
27 $minLines, $caseless, $showLineNumbers, $dots);
28 my (@files);
. . .
31 my ($stars);
32 my ($unique_name);
. . .
35 $caseless = 0;
36 $showLineNumbers = 0;
. . .
38 $stars = "**************************************************";
39 $unique_name = Digest::MD5::md5_hex (rand);
. . .
42 'caseless' => \$caseless,
43 'numbers' => \$showLineNumbers,
. . .
56 print "perl match.pl [-caseless] [-numbers] [-dots] \n" .
57 "\t\t\t[-lines NUMBER] [-output PATH]\n" .
. . .
66 print "perl match.pl [-caseless] [-numbers] [-dots] \n" .
67 "\t\t\t[-lines NUMBER] [-output PATH]\n" .
. . .
96 print OUT substr ($stars, 1, (75 - length ($files[$b]))/2) . " " .
97 $files[$b] . " " .
98 substr ($stars, 1, (75 - length ($files[$b]))/2) . "\n";
99 $lineNumber = 0;
. . .
ready 10:29:19
|
Example 9 - caselessness
The first match is for "noah" and shows what it looks like when nothing matches - just the header line and a ready prompt. In the second match I have added a -c (-caseless) and I get 1 line.
perl match.pl -ma noah -f match.pl ********************************* match.pl ********************************* ready 10:30:00 perl match.pl -ma noah -f match.pl -ca ********************************* match.pl ********************************* # Noah.Davids@stratus.com ready 10:30:08 |
Example 10 - multiple files using star names
This is matching for "noah" in any file ending with .pl in the current directory
perl match.pl -ma noah -f *.pl -ca ******************************** calling.pl ******************************** ******************************** diff_ns.pl ******************************** # Please direct any questions or comments to noah_davids@stratus.com ******************************** genpass.pl ******************************** ********************************* match.pl ********************************* # Noah.Davids@stratus.com ***************************** netstat_filter.pl ***************************** # noah_davids@stratus.com ******************************** pm21line.pl ******************************** # Noah.Davids@stratus.com ********************************** test.pl ********************************** ********************************* test_1.pl ********************************* *********************************** x.pl *********************************** ready 10:30:40 |
Example 11 - multiple -f arguments with star names
This is the same match as example 10 but with 2 sets of files. The first is all *.pl in the current directory and the second is all*.pl in the testing subdirectory.
perl match.pl -ma noah -f *.pl -ca -f testing>*.pl ******************************** calling.pl ******************************** ******************************** diff_ns.pl ******************************** # Please direct any questions or comments to noah_davids@stratus.com ******************************** genpass.pl ******************************** ********************************* match.pl ********************************* # Noah.Davids@stratus.com ***************************** netstat_filter.pl ***************************** # noah_davids@stratus.com ******************************** pm21line.pl ******************************** # Noah.Davids@stratus.com ********************************** test.pl ********************************** ********************************* test_1.pl ********************************* *********************************** x.pl *********************************** ****************************** testing/foo.pl ****************************** # Noah.Davids@stratus.com ready 10:32:50 |
Example 12 - multiple -f arguments with star names and no match
This is the same match as example 11 but no files match the second -f argument
perl match.pl -ma noah -f *.pl -ca -f testing>*.pl ******************************** calling.pl ******************************** ******************************** diff_ns.pl ******************************** # Please direct any questions or comments to noah_davids@stratus.com ******************************** genpass.pl ******************************** ********************************* match.pl ********************************* # Noah.Davids@stratus.com ***************************** netstat_filter.pl ***************************** # noah_davids@stratus.com ******************************** pm21line.pl ******************************** # Noah.Davids@stratus.com ********************************** test.pl ********************************** ********************************* test_1.pl ********************************* *********************************** x.pl *********************************** No files found for testing/*.pl ready 10:34:56 |
Example 13 - using an output file
This is the same match as example 10 but output is directed to the file foo.
perl match.pl -ma noah -f *.pl -ca -o foo ready 10:36:12 d foo %phx_vos#m16_mas>SysAdmin>Noah_Davids>foo 09-10-09 10:36:20 mst ******************************** calling.pl ******************************** ******************************** diff_ns.pl ******************************** # Please direct any questions or comments to noah_davids@stratus.com ******************************** genpass.pl ******************************** ********************************* match.pl ********************************* # Noah.Davids@stratus.com ***************************** netstat_filter.pl ***************************** # noah_davids@stratus.com ******************************** pm21line.pl ******************************** # Noah.Davids@stratus.com ********************************** test.pl ********************************** ********************************* test_1.pl ********************************* *********************************** x.pl *********************************** ready 10:36:20 |
Example 14 - filtering the output from a command
perl match.pl -co 'netstat -statistics' -ma Err -ma 'if '
************************************* *************************************
0 tcpInErrs
0 udpInErrors
0 ipInHdrErrors
0 ipInAddrErrors
0 ipsecOutputENETDOWNErrors
0 ipsecOutputENOSRErrors
0 ipsecOutputDropGetSAErrors
0 ipsecOutputBufferOverflowErrors
247 icmpInErrors
0 icmpOutErrors
if #1 (%phx_vos#loop.m16, localhost), up:
0 ifInErrors
0 ifOutErrors
if #2 (%phx_vos#sdlmuxA.m16.10-5-0.11-5-0, 10.10.1.1, 00 00 a8 40 52 22), up:
0 ifInErrors
0 ifOutErrors
if #3 (%phx_vos#sdlmuxA.m16.10-5-1.11-5-1, 10.20.1.1, 00 00 a8 41 52 22), up:
0 ifInErrors
0 ifOutErrors
if #4 (%phx_vos#sdlmux.m16.10.11-2, phx_vos-m16, 00 00 a8 42 52 22), up:
0 ifInErrors
0 ifOutErrors
if #5 (%phx_vos#sdlmux.m16.11.11-2, 172.16.1.226, 00 00 a8 43 52 22), up:
0 ifInErrors
0 ifOutErrors
if #6 (%phx_vos#sdlmux.m16.11-3, rigel, 00 00 a8 44 52 22), up:
0 ifInErrors
0 ifOutErrors
ready 10:53:02
|
Example 15 - using a command macro to execute multiple commands
You must execute that macro from "sh" If you execute the command directly only the output from the first command is filtered. The output from the other commands is just sent to the terminal
perl match.pl -co 'sh x.cm' -ma 'Device Name' -ma CRC ************************************* ************************************* Device Name: #sdlmuxA.m16.10-5-0.11-5-0 Receive frame discarded, bad CRC : 0 Device Name: #sdlmuxA.m16.10-5-0.11-5-0 Receive frame discarded, bad CRC : 0 Device Name: #sdlmuxA.m16.10-5-1.11-5-1 Receive frame discarded, bad CRC : 0 Device Name: #sdlmuxA.m16.10-5-1.11-5-1 Receive frame discarded, bad CRC : 0 Device Name: #sdlmux.m16.10.11-2 Receive frame discarded, bad CRC : 0 Device Name: #sdlmux.m16.11.11-2 Receive frame discarded, bad CRC : 0 Device Name: #sdlmux.m16.11-3 Receive frame discarded, bad CRC : 0 Device Name: #sdlmux.m16.11-3 Receive frame discarded, bad CRC : 0 ready 12:05:24 d x.cm %phx_vos#m16_mas>SysAdmin>Noah_Davids>x.cm 09-10-09 12:10:26 mst netstat -interface \#sdlmuxA.m16.10-5-0.11-5-0 netstat -interface \#sdlmuxA.m16.10-5-1.11-5-1 netstat -interface \#sdlmux.m16.10.11-2 netstat -interface \#sdlmux.m16.11.11-2 netstat -interface \#sdlmux.m16.11-3 |
# match.pl begins here
#
# match.pl
# version 1.0 08-04-18
# version 1.1 08-05-20 added output switch to send output to a file instead of
# terminal screen
# version 1.2 09-10-09 added commandString argument to filter on command
# output instead of a file
# version 1.3 10-11-26 added disclaimer
# Noah.Davids@stratus.com
#
# The latest version of this macro and documentation can be found at
# http://noahdavids.org/self_published/match.html
#
# This software is provided on an "AS IS" basis, WITHOUT ANY WARRANTY OR ANY
# SUPPORT OF ANY KIND. The AUTHOR SPECIFICALLY DISCLAIMS ANY IMPLIED WARRANTIES
# OF MERCHANTABILITY OR FITNESS FOR ANY PARTICULAR PURPOSE. This disclaimer
# applies, despite any verbal representations of any kind provided by the
# author or anyone else.
#
use strict;
use warnings;
use Getopt::Long;
use Digest::MD5;
my (@match, @except, @inputFile, @outputFile, @commandString,
$minLines, $caseless, $showLineNumbers, $dots);
my (@files);
my ($INFILE);
my ($result, $a, $b, $i, $j, $print, $lineNumber, $lastLinePrinted, $reset);
my ($stars);
my ($unique_name);
$minLines = 0;
$caseless = 0;
$showLineNumbers = 0;
$dots = 0;
$stars = "**************************************************";
$unique_name = Digest::MD5::md5_hex (rand);
$result = GetOptions ('lines=s' => \$minLines,
'caseless' => \$caseless,
'numbers' => \$showLineNumbers,
'dots' => \$dots,
'match=s' => \@match,
'except=s' => \@except,
'file=s' => \@inputFile,
'module=s' => \@match,
'output=s' => \@outputFile,
'command=s' => \@commandString);
if (($result != 1) || (@match == 0) ||
((@inputFile == 0) && (@commandString == 0)))
{
print "\n\nUsage:\n";
print "perl match.pl [-caseless] [-numbers] [-dots] \n" .
"\t\t\t[-lines NUMBER] [-output PATH]\n" .
"\t\t\t-match STRING [-match STRING]* [-except STRING]* \n" .
"\t\t\t[[-file PATH]* | [-command STRING]]\n";
exit;
}
if ((@inputFile > 0) && (@commandString > 0))
{
print "\n\nUsage:\n";
print "perl match.pl [-caseless] [-numbers] [-dots] \n" .
"\t\t\t[-lines NUMBER] [-output PATH]\n" .
"\t\t\t-match STRING [-match STRING]* [-except STRING]* \n" .
"\t\t\t[[-file PATH]* | [-command STRING]]\n";
print "\n\n\t\tYou can only provide files or a command string - not both\n";
exit
}
if (@commandString)
{
system ($commandString [0] . " > " . $unique_name);
if ($? == 0) { $inputFile [0] = $unique_name; }
else {die "could not execute comand " . $commandString[0] . "\n";}
}
if (!@outputFile) {open (OUT, ">&STDOUT");}
else {open (OUT, ">".$outputFile[0]) ||
die "Can't open outfile " . $outputFile[0];}
for ($a = 0; $a < @inputFile; $a++)
{
$inputFile[$a] =~ s/>/\//g;
$inputFile[$a] =~ s/</\.\.\//g;
@files = glob ($inputFile[$a]);
if (@files < 1) {print OUT "\n\nNo files found for " .
$inputFile[$a] . "\n\n";}
for ($b = 0; $b < @files; $b++)
{
open ($INFILE, $files[$b]);
if (@commandString) { $files[$b] = "" }
print OUT substr ($stars, 1, (75 - length ($files[$b]))/2) . " " .
$files[$b] . " " .
substr ($stars, 1, (75 - length ($files[$b]))/2) . "\n";
$lineNumber = 0;
$lastLinePrinted = 0;
while ($_ = <$INFILE>)
{
$lineNumber++;
$print = 0;
if ($caseless)
{
for ($i = 0; $i < @match; $i++) {if (/$match[$i]/i) {$print++;}}
for ($i = 0; $i < @except; $i++) {if (/$except[$i]/i)
{$print = 0;}}
}
else
{
for ($i = 0; $i < @match; $i++) {if (/$match[$i]/) {$print++;}}
for ($i = 0; $i < @except; $i++) {if (/$except[$i]/)
{$print = 0;}}
}
if ($print > 0)
{
if (($lastLinePrinted + 1 < $lineNumber) && $dots)
{print OUT " . . . \n";}
if ($showLineNumbers) {print OUT $lineNumber . "\t";}
print OUT $_;
$lastLinePrinted = $lineNumber;
for ($j = 2; $j <= $minLines; $j++)
{
$_ = <$INFILE>;
if (defined ($_))
{
$lineNumber++;
if ($showLineNumbers) {print OUT $lineNumber . "\t";}
print OUT $_;
$lastLinePrinted = $lineNumber;
$reset = 0;
if ($caseless)
{
for ($i = 0; $i < @match; $i++)
{if (/$match[$i]/i) {$reset++;}}
for ($i = 0; $i < @except; $i++)
{if (/$except[$i]/i) {$reset = 0;}}
}
else
{
for ($i = 0; $i < @match; $i++)
{if (/$match[$i]/) {$reset++;}}
for ($i = 0; $i < @except; $i++)
{if (/$except[$i]/) {$reset = 0;}}
}
if ($reset > 0) {$j = 1;}
}
}
}
}
if (($lastLinePrinted + 1 < $lineNumber) && $dots)
{print OUT " . . .\n";}
close $INFILE;
}
}
#
if (@commandString) { unlink ($unique_name); }
#
# match.pl1 ends here
|
Send comments and suggestions