A Better Tshark follow stream

Blue Bar separator

Wireshark has a very nice "follow stream" feature where the data from a stream is displayed in a nicely color coded window.


Figure 1 - The Follow TCP Stream window from Wireshark

Unfortunately Tshark's output is not quite as nice. I find it hard to follow which host sent what message without the color coding.

tshark -r test3.pcapng -q -z follow,tcp,ascii,1

===================================================================
Follow: tcp,ascii
Filter: tcp.stream eq 1
Node 0: 192.168.1.200:32885
Node 1: 50.62.107.1:80
284
GET / HTTP/1.1
Host: noahdavids.org
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Firefox/31.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Connection: keep-alive


        1448
HTTP/1.1 200 OK
Date: Tue, 03 Mar 2015 15:52:17 GMT
Server: Apache
Accept-Ranges: bytes
Vary: Accept-Encoding
Content-Encoding: gzip
Content-Length: 1469
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Content-Type: text/html

...........VMs.F..g&..e.If"2..t.J..v\k.8.X...gE....]vwiZ....+JN.I..H...... &?..
;^|.|..oj..pt>?.d.e...Yv.8....o.a......)...u...H.?z.(..o_gY..i........{r7...r.w
..O.....`c.E.W.Fz.d?..t.n..F{....[.@..&^.....
y%..~.+]....{.{.d..+_.......;U8..........t.1^.I...@.q....kX..v....&..+.2..[..O.
..Or.#-]U..P.k.O.gW.J].V.5(...N.....C..W....>.P.
.......Ka..*.`.ir....*.V.r1~......-.HXp..Vxc.a6@........le9M
..KC-.+..h..Sc....I&f..V.+."....YCK.....+`.....|%.
.A-...%..`..
.6.\nU.7......#.'G.....i......[.=..\<.......+4.
...]b.zT+-..W9..5..j".V[....7D....
;.w..]+._..n.j.
9.....O.....E..... ..%.s.$...1E'?...`....l.*.-.c0.+..&.s....o9.y..i...b.....[..
.....O....i..\-IC?..p-I,.%W...y...T?........1k.........b...v,.,_.t.5/[T...>....
...\....t..X..(>cW..M...l..Uel,9V..f....j@k.S$.9(..].#.M..b...w..h.Ds..L4d.`\..
8WK&....G..}...|i..e2.<9e.Tm.#zp~MCw...H....,BG.|.7.7.z...tu.}.......y,.m...!..
"....L/..}..X..RK.z.O.....w..e..kk"lc.2.w3.....#7....ea....*.,)......./...=?.eG
.._...^..`.
U..........e!t.h..G44.J....@.
G.t.Wo=.L.~..A3.3..S.,..Q...x5...<P....?.lI+!..p7'Q.n......[..-z.?...EK......M.
.
        265
).F.."XP....i(.-....K*b.$..a...S.|.RiM.,..[..6Q..(#.#..h.C..hmi9...j.....m."vE.
.z8..N.\..|x.....F<.l.~m:h8..R4.#...38.E...x.{.w........[......pa.j..Sa..<.{...
.W.....6P..7....W.......=Kq.If_..WKs......6]3.hHwj..I-K.f..F...z%.R..B..n..9..y
. 6.7.Q6<.2Z................
305
GET /images/bluebar.gif HTTP/1.1
Host: noahdavids.org
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Firefox/31.0
Accept: image/png,image/*;q=0.8,*/*;q=0.5
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Referer: http://noahdavids.org/
Connection: keep-alive


        1350
HTTP/1.1 200 OK
Date: Tue, 03 Mar 2015 15:52:17 GMT
Server: Apache
Last-Modified: Tue, 18 May 2010 16:27:57 GMT
ETag: "3f1-486e0d2f65d40"
Accept-Ranges: bytes
Content-Length: 1009
Cache-Control: max-age=5184000
Expires: Sat, 02 May 2015 15:52:17 GMT
Keep-Alive: timeout=5, max=99
Connection: Keep-Alive
Content-Type: image/gif

GIF89a@...w..!..MSOFFICE9.0.....msOPMSOFFICE9.0../V..!..MSOFFICE9.0.....cmPPJCm
p0712....H.s..!.....P.,....@..............................................~..{.
.y..i~.e~._u.]r.T_sOx.Nw.Nv.Lx.Kv.Ju.Eh.Dd.DM]DLTCg.Ce.AO.=X.<F`;T.:AJ.G..C|.@g
-27.R..G..Z..A..A..U..S..Q..1o.%6.!A.!/........................................
...............................................................................
..................................... ...P.....................................
...................'#................7+-...............71/-4#..................
........
.H......*\.NA...ZHlQ...#.3j...... C..I....(S.\.....0c..I.......@Q....I...J....H
.*].....P.J.J....X.j......`.
...C
..[.h.D...p....K....x............L......+^....%1X..a.G...3k.......C..M.....S.^.
.....c..M..m.=(S............8....#_..9....C...:....c...;....../.<....._o.H..>i.
.................(....h...&....6....F(...Vh......|-..B...(..$.h...|.a.....} .(.
.4.h.....!.:.@B.8.)..D....@...;
===================================================================
Figure 2 - Tshark follow command

So I came up with my own shell script which labels each message with the source and destination AND color codes them. The shell script requires a very simple Perl script to convert the hex formatted data output by the Tshark command into ASCII characters.

./programs/sh/tshark-follow-stream.sh test3.pcapng 32885 80                    

192.168.1.200 32885  --->  50.62.107.1 80
GET / HTTP/1.1..Host: noahdavids.org..User-Agent: Mozilla/5.0 (X11; Linux x86_6
4; rv:31.0) Gecko/20100101 Firefox/31.0..Accept: text/html,application/xhtml+xm
l,application/xml;q=0.9,*/*;q=0.8..Accept-Language: en-US,en;q=0.5..Accept-Enco
ding: gzip, deflate..Connection: keep-alive....

50.62.107.1 80  --->  192.168.1.200 32885
HTTP/1.1 200 OK..Date: Tue, 03 Mar 2015 15:52:17 GMT..Server: Apache..Accept-Ra
nges: bytes..Vary: Accept-Encoding..Content-Encoding: gzip..Content-Length: 146
9..Keep-Alive: timeout=5, max=100..Connection: Keep-Alive..Content-Type: text/h
tml...............VMs.F..g&..e.If"2..t.J..v\k.8.X...gE....]vwiZ....+JN.I..H....
.. &?..;^|.|..oj..pt>?.d.e...Yv.8....o.a......)...u...H.?z.(..o_gY..i........{r
7...r.w.../....p....n..O.....`c.E.W.Fz.d?..t.n..F{....[.@..&^......y%..~.+]....
{.{.d..+_.......;U8..........t.1^.I...@.q....kX..v....&..+.2..[..O...Or.#-]U..P
.k.O.gW.J].V.5(...N.....C..W....>.P.........Ka..*.`.ir....*.V.r1~......-.HXp..V
xc.a6@........le9M...KC-.+..h..Sc....I&f..V.+."....YCK.....+`.....|%...A-...%..
`.....|x.._.....6.
U.7......#.'G.....i......[.=..\<.......+4.....]b.zT+-..W9..
5..j".V[....7D.....;.w..]+._..n.j..9.....O.....E..... ..%.s.$...1E'?...`....l.*
.-.c0.+..&.s....o9.y..i...b.....[........q'T-...s.Djz..."2..)..i.a)....WN...+~N
M]....f...i......#..z(Q........O....i..\-IC?..p-I,.%W...y...T?........1k.......
..b...v,.,_.t.5/[T...>......'3....kK).........C..%..\....t..X..(>cW..M...l..Uel
,9V..f....j@k.S$.9(..].#.M..b...w..h.Ds..L4d.`\..8WK+..........u.&....G..}...|i
..e2.<9e.Tm.#zp~MCw...H....,BG.|.7.7.z...tu.}.......y,.m...!.."....L/..}..X..RK
.z.O.....w..e..kk"lc.2.w3.....#7....ea....*.,)......./...=?.eG.._...^..`..U....
......e!t.h..G44.J....@..G.t.Wo=.L.~..A3.3..S.,..Q...x5...<P....?.lI+!..p7'Q.n.
.....[..-z.?...EK......M..

50.62.107.1 80  --->  192.168.1.200 32885
).F.."XP....i(.-....K*b.$..a...S.|.RiM.,..[..6Q..(#.#..h.C..hmi9...j.....m."vE.
.z8..N.\..|x.....F<.l.~m:h8..R4.#...38.E...x.{.w........[......pa.j..Sa..<.{...
.W.....6P..7....W.......=Kq.If_..WKs......6]3.hHwj..I-K.f..F...z%.R..B..n..9..y
. 6.7.Q6<.2Z................

192.168.1.200 32885  --->  50.62.107.1 80
GET /images/bluebar.gif HTTP/1.1..Host: noahdavids.org..User-Agent: Mozilla/5.0
 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Firefox/31.0..Accept: image/png,im
age/*;q=0.8,*/*;q=0.5..Accept-Language: en-US,en;q=0.5..Accept-Encoding: gzip,
deflate..Referer: http://noahdavids.org/..Connection: keep-alive....

50.62.107.1 80  --->  192.168.1.200 32885
HTTP/1.1 200 OK..Date: Tue, 03 Mar 2015 15:52:17 GMT..Server: Apache..Last-Modi
fied: Tue, 18 May 2010 16:27:57 GMT..ETag: "3f1-486e0d2f65d40"..Accept-Ranges:
bytes..Content-Length: 1009..Cache-Control: max-age=5184000..Expires: Sat, 02 M
ay 2015 15:52:17 GMT..Keep-Alive: timeout=5, max=99..Connection: Keep-Alive..Co
ntent-Type: image/gif....GIF89a@...w..!..MSOFFICE9.0.....msOPMSOFFICE9.0../V..!
..MSOFFICE9.0.....cmPPJCmp0712....H.s..!.....P.,....@..........................
....................~..{..y..i~.e~._u.]r.T_sOx.Nw.Nv.Lx.Kv.Ju.Eh.Dd.DM]DLTCg.Ce
.AO.=X.<F`;T.:AJ.G..C|.@g-27)<s(5J&2C&*+%2I%1@$2S$2Q#1T#.:!1_!,7.-T.,R. !.*N...
................R..G..Z..A..A..U..S..Q..1o.%6.!A.!/............................
...............................................................................
................................................. ...P.........................
...............................'#................7+-...............71/-4#......
......................H......*\.NA...ZHlQ...#.3j...... C..I....(S.\.....0c..I..
.....@Q....I...J....H.*].....P.J.J....X.j......`.....C...[.h.D...p....K....x...
.........L......+^....%1X..a.G...3k.......C..M.....S.^......c..M..m.=(S........
....8....#_..9....C...:....c...;....../.<....._o.H..>i..................(....h.
..&....6....F(...Vh......|-..B...(..$.h...|.a.....} .(..4.h.....!.:.@B.8.)..D..
..@...;
Figure 3 - my "improved" follow stream command

I have tested this under:

tshark-follow-stream.sh

Usage

         tshark-follow-stream.sh FILE PORT1 PORT2

FILE Is the path to a packet trace file suitable for Tshark to process

PORT1 The service port number of the stream.

PORT2 The other port number.

The script tells Tshark to interpret the segments using the specified TCP ports as the ECHO protocol, which basically means no interpretation. The TCP data is just a stream of bytes. It uses a filter of "tcp.port == PORT1 && tcp.port== PORT2 && echo.data" which filters out any segment without TCP data, that is the TCP control segments and segments containing only ACKs. The two ports should be enough to select only 1 stream. If port numbers are reused or multiple hosts use the same port pairs you may need to add an IP address to the filter. I wanted to keep the arguments as simple as possible for the typical case.

The echo.data format is XX:XX:XX... where XX represents the hex value of the byte. Once Tshark outputs a segment the "XX:XX:..." format is processed by the tshark-data-2-text.pl Perl script to generate a string of human readable characters. Any "XX" value that represents a non readable character is output as a "dot".

Segments from PORT1 to PORT2 are color coded in RED while segments from PORT2 to PORT1 are color coded in BLUE. If you do not like the colors search the internet (I hesitate to say "Google") for "terminal color codes Linux" to get other color codes.

#! /bin/sh
# tshark-follow-stream.sh begins
#
# This macro replies on the tshark-data-2-text.pl Perl script to display the
# actual data in the segment double check the path that the script is using
# and make any necessary changes.
#
# Usage
# tshark-follow-stream.sh FILE PORT1 PORT2
# Where
#  FILE is a file that can be interpreted by Tshark
#  PORT1 is the server port number of the stream
#  PORT2 is the other port number of the stream
#
# Example
#   tshark-follow-stream.sh test3.pcapng 80 32885
#
# Version 1.0 March 8, 2015
#
# See http://noahdavids.org/self_published/tshark-follow-stream.html
#
# This software is provided on an "AS IS" basis, WITHOUT ANY WARRANTY OR ANY
# SUPPORT OF ANY KIND. The AUTHOR SPECIFICALLY DISCLAIMS ANY IMPLIED WARRANTIES
# OF MERCHANTABILITY OR FITNESS FOR ANY PARTICULAR PURPOSE.  This disclaimer
# applies, despite any verbal representations of any kind provided by the
# author or anyone else.
#
FILE=$1
PORT1=$2
PORT2=$3

if [ $# -ne 3 ]
   then echo -e "Usage:\n\ttshark-follow-stream FILE STREAM PORT1 PORT2 IP"
        echo -e "\t\tFILE is a file that can be interpreted by Tshark"
        echo -e "\t\tPORT1 is the server port number of the stream"
        echo -e "\t\tPORT2 is the other port number of the stream"
        echo -e "\t\tExample:"
        echo -e "\t\t  tshark-follow-stream test3.pcapng 80 32885"
        exit
fi
#
tshark -r $FILE -d tcp.port==$PORT1,echo -d tcp.port==$PORT2,echo \
  -R "tcp.port==$PORT1 && tcp.port==$PORT2 && echo.data" \
  -T fields -e ip.src -e ip.dst -e tcp.srcport -e tcp.dstport -e echo.data | \
  while read src dst sport dport data
do
if [[ "$sport" == "$PORT1" && "$dport" == "$PORT2" ]]
    then echo -e "\E[31m"  # 31 indicates red
elif [[ "$sport" == "$PORT2" && "$dport" == "$PORT1" ]]
    then echo -e "\E[34m"  # and 34 indicates blue
fi
echo $src $sport  " ---> " $dst $dport
echo -ne $(echo $data | perl tshark-data-2-text.pl)
echo -e "\E[0m"            # and this resets the color
done
#
# tshark-follow-stream.sh ends here

tshark-data-2-text.pl

# tshark-data-2-text.pl begins
#
# This script takes a stream of characters in the format XX:XX:XX:XX from
# STDIN where each XX is a hex value between 0 and FF. Values between
# decimal 32 (space) and 126 (~) inclusive are written to STDOUT, values
# outside that range are written to STDOUT as a dot (.). It is assumed that
# the output stream will be processed by the command "echo -e" so the backslash
# character (\) is written twice. Every 79 characters (the second backslash is
# not counted) a newline is inserted into the output stream.
#
# This script was written to convert Tshark echo.data into readable text
# for the script tshark-follow-stream-sh.
#
# Version 1.0 March 8, 2015
#
# See http://noahdavids.org/self_published/tshark-follow-stream.html
#
# This software is provided on an "AS IS" basis, WITHOUT ANY WARRANTY OR ANY
# SUPPORT OF ANY KIND. The AUTHOR SPECIFICALLY DISCLAIMS ANY IMPLIED WARRANTIES
# OF MERCHANTABILITY OR FITNESS FOR ANY PARTICULAR PURPOSE.  This disclaimer
# applies, despite any verbal representations of any kind provided by the
# author or anyone else.
#
use POSIX;
#
$count = 0;
while (<STDIN>)
   {
   $lineLength = length ($_);
   for ($i = 0; $i < $lineLength; $i=$i+3) {
       $charNumber = strtol (substr ($_, $i, 2), 16);
       if (($charNumber < 32) || ($charNumber > 126))
           { $charNumber = 46; }
       $character = chr ($charNumber);
       print $character;
       if ($charNumber == 92)  # back-slash in the character stream needs
          { print $character; } # to be escaped with a back-slash
       $count = $count + 1;
       if ($count > 78)
          {
          print "\\n";
          $count = 0;
          }
       }
   }
# tshark-data-2-text.pl ends here



Blue Bar separator
This page was last modified on 2015-03-08
mailbox Send comments and suggestions
to noah@noahdavisds.org