wuetender-junger-mann.de

Quite often I want to pipe the content of multiple files into a command line utility. An example would be to count the lines of sql in my project. This is another case, where xargs comes in handy:

find . -name "*.sql" | xargs cat | wc -l

I recently had the pleasure of supporting a new system throughout its first month of production. This was a good opportunity to refresh my command line skills. As it happened I spent a lot of time looking at log files trying to figure out what happened to the productions system. I figured, that a graphical representation of the events would be nice and started using gnuplot.

First I started out with a bunch of bash scripts, using what your usual unix installation provides, but then I actually came up with some groovy scripts to provide better abstractions. A log file generally looks somewhat like this:

04/01/1970 07:55:13 garbage
04/01/1970 09:27:48 Event 2
04/01/1970 10:01:28 garbage
04/01/1970 10:38:30 garbage
04/01/1970 10:48:36 garbage
04/01/1970 10:51:58 Event 2
04/01/1970 11:03:45 garbage
04/01/1970 11:34:03 Event 1
04/01/1970 12:24:33 garbage
05/01/1970 04:35:50 ERROR

There is a lot of garbage plus some events we might be interested in. It allows to specify events, e.g. by providing a regexp:

Event EVENT1 = new RegExEvent("Event 1", ~/Event 1/)
Event EVENT2 = new RegExEvent("Event 2", ~/Event 2/)
Event ERROR = new RegExEvent("Error", ~/ERROR/)

The next step is newing up a TimeLineVisualizer on these events and passing in a stream with the actual log:

def logFile = new File("test.log");
 
logFile.withInputStream {InputStream stream ->
  def visualizer = new TimeLineVisualizer([
          EVENT1,
          EVENT2,
          ERROR
  ]);
  visualizer.visualize(stream)
}

If you have the gnuplot binary on your path this will yield something like this:

Also in some cases you would like to know which time of day events are most likely to happen. For producing histograms I created another visualizer (which currently takes only one event).

logFile.withInputStream {InputStream stream ->
  def visualizer = new HistogramVisualizer(EVENT2, HistogramVisualizer.HOUR_OF_DAY_BINS)
  visualizer.visualize(stream)
}

For the example log file, which unfortunately has an even distribution of events, we get this:

The cool thing about gnuplot is, that you can actually run these things in a cron job to produce daily reports (and mail them to the appropriate people) or on a continuous integration server to visualise how the system is being exercised by the test suite.

Every now and then even the über-geek can’t avoid the use of non-command-line-tools. Currently I am doing a bit of production support work, so tail, grep, and awk are my friends. However people expect emails, excel spreadsheets, and similar stuff. So what to do? Well use the best of both worlds pipe your output into the clipboard and vice versa. Here is how it works for different platforms:

Cygwin

If you are a software developer or any other serious user of computing gear on windows get cygwin!

ls | putclip # Will copy the output of ls to the clipboard getclip | grep "ERROR 500" # Will grep on the contents of the clipboard

OSX

ls | pbcopy # Will copy the output of ls to the clipboard pbpaste | grep "ERROR 500" # Will grep on the contents of the clipboard

Linux

ls | xsel --clipboard # Will copy the output of ls to the clipboard xsel --clipboard | grep "ERROR 500" # Will grep on the contents of the clipboard
Alternatively you might want to try xclip.

QuickSilver

And while we are at it, if you are a QuickSilver user you should have a look at the qs commandline tool, which lets you pipe contents into qs or open qs on files.
qs mylovelyfile.ext # opens quicksilver on a file (useful for sending it by mail)

And of course I shamelessly stole that stuff from all over the place, e.g. there.

fortune | xargs cowsay

Beeing budget-driven you have to have the right tools. The simplest thing I could come up with are the code scales, that tell you what you have got in your workspace:

 (svn diff |grep -e "^\+[^\+]"|wc -l;svn diff|grep -e "^-[^-]"|wc -l;echo "-";echo "p")|dc

It once again proves the point that real programmers use unix and love RPN.