Pattern Parser

The Pattern Parser uses a conversion pattern to read a log file. If you are familiar with the Apache Log4 libraries, you are probably already familiar with Pattern Layouts. It is basically the same in LogViewPlus. In fact...

If you use a pattern layout for writting your log file, you can probably use this as your conversion pattern for parsing your log file.

To find out more about conversion patterns, let's start with a simple log entry:

2014-09-13 10:50:14,125 [Thread 1] INFO MyApp - My message...

Here we have a simple entry made up of five parts: date, thread, log level, logger, and the message. We need a way to tell LogViewPlus about each part of our log entry as well as how to tell when one part ends and the next begins. To do this, we are going to use patterns.

Now, back to our log entry. Let's break our log entry down piece by piece:

Field
Example
Conversion Specifier
Date
2014-09-13 10:50:14,125
%d
Literal
" ["
" ["
Thread
Thread 1
%t
Literal
"] "
"] "
Priority
INFO
%p
Literal
" "
" "
Logger
MyApp
%c
Literal
" - "
" - "
Message
My Message...
%m
Literal
New Line.
%n

If we put all of the conversion specifiers together we have:

%d [%t] %p %c - %m%n

If the sample log entry above matched your log entries, you could use the pattern we created to parse your log files. Unfortunately, this is probably not the case as every log file is different. You will need to figure out the conversion pattern that works for your log files and this may require a bit of trial and error. The Parser Wizard can help you create and test an appropriate conversion pattern.

Once you have decided on the appropriate conversion pattern, you will need to provide that as an argument to the Pattern Parser.

Conversion patterns can be a bit tricky at first, so here is a cheat sheet. For getting the most out of LogViewPlus, you only really need to know 7 different types of conversion specifiers.

Field
Specifier
Notes
Date
%d
Dates can be tricky because there are so many ways they can be represented. See Date Patterns for more.
Thread
%t
Thread names can contain spaces, so you need a non-spaced literal seperating the thread from other fields. For example, brackets in "[my thread]".
Priority
%p
DEBUG, INFO, WARN, ERROR, etc...
Logger
%c
The logger responsible for writing the log entry. Very helpful when filtering.
Message
%m%n
When using the PatternParser, these two conversion specifiers almost always go together.
Any Word
%s
Cheat #1 - match any string without spaces.
Multiple Words
%S
Cheat #2 - match any array of strings spaces included. Note this field is upper-case.

Check out Advanced Specifiers for the full list of supported conversion specifiers.

Why are %s and %S cheats? Two reasons: First, they are specific to LogViewPlus and not supported by standard logging frameworks like Apache Log4. Second, the %s specifiers can optionally take a parameter which specifies the name of the column which should display the field. To do this, you just need to enclose the column name in curly brackets. For example: %s{My Column Name}. This ability to take any field and convert it into a LogViewPlus column makes these specifiers extremely powerful. Note that the %s specifiers are the only conversion specifiers which do not have a predefined column name.

If the optional column name is not specified, then LogViewPlus will not know how to display the field. You will still be able to use LogViewPlus to search for this text, but it will not be shown in a dedicated column in the log entry grid.

You can find out more about %s specifiers by seeing them used to parse an IIS log file.

Now, just for fun, lets go through a few quick examples using just the specifiers listed above. For simplicity, these examples will use timestamps only. Dates are discusssed in detail in the Date Specifier section.

Log Entry:
07:36|My message...
Pattern:
%d|%m%n
Notes:
The pipe character '|' is used to seperate date and message.
Log Entry:
10:50 [Thread 1] INFO property1 - My message...
Pattern:
%d [%t] %p %s - %m%n
Notes:
The 'any word' specifier '%s' is used to skip over the unknown field 'property1'.
Log Entry:
10:50 [Thread 1] INFO ndc1 ndc2 ndc3 - My message...
Pattern:
%d [%t] %p %S - %m%n
Notes:
Skipping over multiple fields with the %S specifier.
Log Entry:
10:50 Thread 1 - My message...
Pattern:
%d %t - %m%n
Notes:
Multi-word thread name parseable thanks to unique literal ' - '.
Log Entry:
10:50|240 1 I 2 LEVEL: Info 3
Pattern:
%d|%S LEVEL: %p %m%n
Notes:
Advanced literal ' LEVEL: ' used to specify a field. Everything prior to the literal is skipped with the multi-word specifier '%S'.
Log Entry:
[10:50] [notice] Apache/1.3.29 (Unix) server configured -- resuming operations
Pattern:
[%d] [%p] %s (%s) %S -- %m%n
Notes:
Good use of literals to seperate strings. Ignoring non-pertinent information.

To find out more about conversion patters, please see Conversion Specifiers.


Multi-Patterns

It is very common to have one logging format per log file. However, this is not a hard requirement. It is possible to have multiple logging formats in a single log file.

That's why the PatternParser allows the possibility of multiple conversion patterns, one per line, as can be seen in the screenshot below:

Note that multi-patterns cannot be configured using the Parser Wizard. You must use manual parser configuration instead.

When using multiple conversion patterns, it's important to keep a few things in mind. First, using multiple patterns will always be slower than using a single pattern. This is because the parser will expect a degree of error when parsing the log file and a single log line will often be parsed multiple times.

When using multi-patterns the first pattern which successfully parses the log line will be used. Therefore, it is best to put more generic patterns which match log fields more broadly at the bottom of the pattern list. For example, '%d %m%n' would match most log entries and therefore should be placed near the bottom.

Note that patterns which span mutliple lines, such as those which declare '\n' as part of the static text, are not supported in multi-patterns.

Finally, when using multi-patterns, it is important that the individual patterns are distinct. For example, if you were using two patterns and every log entry could be parsed by either pattern successfully then LogViewPlus will not know which pattern to use. 'Successful' in this case means that the log entry is parsed without error and does not necessarilly mean that data is placed in the correct column.

< >