Pattern Parser

The Pattern Parser uses a conversion pattern to read a log file.  If you are familiar with the Apache Log4 libraries, you are probably already familiar with Pattern Layouts.  It is basically the same in LogViewPlus.  In fact, if you use a pattern layout for writing your log file, you can probably use this same pattern for parsing your log file.

To find out more about conversion patterns, let's look at a simple log entry:

2014-09-13 10:50:14,125 [Thread 1] INFO MyApp - My message...

This log entry is made up of five parts: date, thread, log level, logger, and the message.  We need a way to tell LogViewPlus about each part of our log entry as well as how to tell when one part ends and the next begins.  To do this, we are going to use patterns. 

Now, back to our log entry.  Let's break it down piece by piece:

Field
Example
Conversion Specifier
Date
2014-09-13 10:50:14,125
%d
Literal
" ["
" ["
Thread
Thread 1
%t
Literal
"] "
"] "
Priority
INFO
%p
Literal
" "
" "
Logger
MyApp
%c
Literal
" - "
" - "
Message
My Message...
%m
Literal
New Line.
%n

If we put all of the conversion specifiers together we have:

%d [%t] %p %c - %m%n

If the sample log entry above matched your log entries, you could use the pattern we created to parse your log files.  Unfortunately, this is probably not the case as every log file is different.  You will need to figure out the conversion pattern that works for your log files and this may require a bit of trial and error.  The Parser Wizard can help you create and test an appropriate conversion pattern. 

Once you have decided on the appropriate conversion pattern, you will need to provide that as an argument to the Pattern Parser.

Conversion patterns can be a bit tricky at first, so here is a cheat sheet.  For getting the most out of LogViewPlus, you only really need to know 7 different types of conversion specifiers.

Field
Specifier
Notes
Date
%d
Dates can be tricky because there are so many ways they can be represented.  See Date Patterns for more.
Thread
%t
Thread names can contain spaces, so you need a non-spaced literal separating the thread from other fields.  For example, brackets in "[my thread]".
Priority
%p
DEBUG, INFO, WARN, ERROR, etc...
Logger
%c
The logger responsible for writing the log entry.  Very helpful when filtering.
Message
%m%n
When using the Pattern Parser, these two conversion specifiers almost always go together.
Any Word
%s
Cheat #1 - match any string without spaces.
Multiple Words
%S
Cheat #2 - match any array of strings spaces included.  Note this field is upper-case.

Check out Advanced Specifiers for the full list of supported conversion specifiers.

Why are %s and %S cheats?  Two reasons:  First, they are specific to LogViewPlus and not supported by standard logging frameworks like Apache Log4.   Second, the %s specifiers can optionally take a parameter which specifies the name of the column which should display the field.  To do this, you just need to enclose the column name in curly brackets.  For example: %s{My Column Name}.  This ability to take any field and convert it into a LogViewPlus column makes these specifiers extremely powerful.  The %s specifiers are the only conversion specifiers which do not have a predefined column name.

If the optional column name is not specified, then LogViewPlus will not know how to display the field.  You will still be able to use LogViewPlus to search for this text, but it will not be shown in a dedicated column in the log entry grid.

To find out more, we have created an example walk-through using %s specifiers to parse an IIS log file. 

Finally, there are two additional special characters which can be used to process white space. 

White Space Field
Representation
Notes
Newline
\n
Represents a hard return which should be processed before processing continues.
Tab
\t
Represents a tab separator.  Using this character sequence is optional, but it make help to make your pattern easier to read.

Now let's go through a few quick examples using just the specifiers listed above.  For simplicity, these examples will only use timestamps that can be automatically detected.  Dates are discussed in detail in the Date Specifier section.  The string columns will also be declared without column names. 

 
Sample Log Entries and Matching Patterns
Log Entry:
07:36|My message...
Pattern:
%d|%m%n
Notes:
The pipe character '|' is used to separate date and message.
 
 
Log Entry:
10:50 [Thread 1] INFO property1 - My message...
Pattern:
%d [%t] %p %s - %m%n
Notes:
The 'any word' specifier '%s' is used to skip over the unknown field 'property1'.
 
 
Log Entry:
10:50 [Thread 1] INFO ndc1 ndc2 ndc3 - My message...
Pattern:
%d [%t] %p %S - %m%n
Notes:
Skipping over multiple fields with the %S specifier.
 
 
Log Entry:
10:50 Thread 1 - My message...
Pattern:
%d %t - %m%n
Notes:
The multi-word thread name can be parsed thanks to unique literal ' - '.
 
 
Log Entry:
10:50|240 1 I 2 LEVEL: Info 3
Pattern:
%d|%S LEVEL: %p %m%n
Notes:
Advanced literal ' LEVEL: ' used to specify a field.  Everything prior to the literal is skipped with the multi-word specifier '%S'.
 
 
Log Entry:
[10:50] [notice] Apache/1.3.29 (Unix) server configured -- resuming operations
Pattern:
[%d] [%p] %s (%s) %S -- %m%n
Notes:
Good use of literals to separate strings.  Ignoring non-pertinent information.
 
 
Log Entry:
[10:50] [notice] Apache/1.3.29 (Unix) server configured.
Resuming operations.
Pattern:
[%d] [%p] %s (%s) %S\n%m%n
Notes:
Same as above, but with a hard return expected before the message processing starts.

To find out more about conversion patters, please see Conversion Specifiers.

Finally, please note that it is possible for a log file to have multiple patterns.  To find out more about parsing log files with multiple patterns, please see Multi Patterns.


< >