Log Message Parsers
This chapter has discussed parsing log files into log entries using regular expressions or conversion specifiers. Log entries often contain a message which may contain a significant amount of data. Because log entries are parsed into an envelope and message, it can be difficult to extract data from the message itself. This is especially true when a log file contains a lot of variation between log entry messages.
To solve this problem, LogViewPlus employs a two stage parsing process. First, the log entries are parsed using the techniques discussed in this chapter. Once the log entry has been parsed, the log message will be parsed as a separate step. By default log entry messages will be parsed using an automatic message parse, but you can use the Parse Message Filter to configure one of three options: automatic parse, pattern parse or regex parse. These options are discussed below.
Once configured, message parsers can be associated with a log file parser configuration and will be saved as Message Parser Settings. A log file can contain multiple message parsers which you can manually configure using the Parse Message Filter. After creating or editing a message parser, all log files using the target configuration will need to be refreshed.
Information extracted from a parsed message will be displayed in the Log Entry Grid when the parse message filter is applied. When viewing parsed messages, the Message column will be temporarily removed. Extracted information can also be used in Reports & Dashboards.
Automatic Message Parse
Automatic message parsing occurs when no configured message parser has been found which matches the current log message. This is means that all configured log message parsers must be attempted before the decision to use an automatic parse can be made.
An automatic parse will scan the log message for text it finds interesting and attempt to extract this information. Examples of interesting text include:
1. Numbers
2. Words in all caps.
3. Words which contain numbers or symbols.
4. Text in brackets, parenthesis or quotes.
5. Text after a colon.
Information extracted from an automatic parse will usually be given a generic column name. For example, Column 1, Column 2, etc. However, a column name may be found in the data if a key/value pair is detected. LogViewPlus may detect a key/value pair when:
1. The key and value are separated by an equals sign. For example: key=value.
2. The key and value are separated by a colon. For example: key: value.
Occasionally, an automatic parse may result in an excessive amount of information. For example, if a log entry contains an XML or JSON statement. In these scenarios, CPU and memory is need to process log message data which may not ultimately be needed. To limit this kind of excessive parsing, an automatic parse will mark any messages with more than 10 parameters as advanced messages with 'Adv. Message'. These message are best processed manually.
Automatic message parsing is always on a best effort basis. Usually, we can obtain useful results using the heuristics described above, but sometimes you may need to manually extract the target information. This can be done using either pattern or regex message parsing discussed below.
Pattern Message Parse
A pattern log message parser uses string conversion specifiers to define a parsing configuration. The generated configuration pattern will be very similar to the configuration generated when using the Pattern Parser but only string conversion specifiers should be used.
When a manual message parse is needed, we recommend using the pattern parser instead of the regex parser. The pattern parse is generally significantly more performant. This can be particularly important when multiple log message parsers need to be associated with the log file configuration.
Pattern parse messages can only be configured by using the Parse Message Filter. Existing message parsers are managed in the Message Parser Settings.
Regex Message Parse
A regex log message parser will use a regular expression to extract information from the log entry message. LogViewPlus uses Microsoft's regex parser internally, so only .NET regular expressions are supported.
As discussed above, the pattern message parser is recommend when a manual parse is needed. However, the regex parser can add significant value to users who are already familiar with regular expressions.
Regex parse messages can only be configured in by using the Parse Message Filter. Existing message parsers are managed in the Message Parser Settings.