LogViewPlus Support » LogViewPlus Support » Help & Support » Help parsing

Help parsing

PIDtuner — Fri, 23 Jun 2023 09:41:14 GMT

If I can get around this, I am definetively getting my license.
So I have an unfortunate log with entries as follow:
[code]22/06/23-09:49:58.007050 Thread INFO read 4 bytes from sock 234. Pending: 0 errno=0 [Comms.cpp:517]
22/06/23-09:49:58.007064 Thread INFO Read from server: RpcGlobalCallback. con: sock=234 [Rpc.cpp:1060]
22/06/23-09:49:58.007072 Thread INFO Prepare for read 4 bytes: [sock=234] [Comms.cpp:359]
22/06/23-09:49:58.007102 Thread INFO No more action Connection::handleConnectionEvent sock = 234. mState = 1 [Comms.cpp:590]
22/06/23-09:49:58.007108 Thread INFO Epolled: 9 flags = EPOLLIN [Comms.cpp:192][/code]
I am using the following pattern:
[code]%d{dd/MM/yy-%H:mm:ss.ffffff} %t %p %m [%F:%L]%n[/code]
But I keep getting bad lines like the highlighted one below, where the File Name column is wrong:

RE: Help parsing

LogViewPlus Support — Fri, 23 Jun 2023 09:41:14 GMT

If you need the additional information, you can parse it out with something like;
[code]%d{dd/MM/yy-%H:mm:ss.ffffff} %t %p %m [%s{Key}=%s{Value}] [%F:%L]%n[/code]
The specifier [%s{Key}=%s{Value}] will extract the information into new Key and Value columns. I left this out earlier because it wasn't clear the information was needed and it makes the pattern a little more complicated. Sorry for the confusion.

We have spent of lot of time optimizing the LogViewPlus pattern parser for performance and flexibility. The model we have decided on for this parser is forward-only, but you are absolutely right that other approaches may make more sense in certain scenarios. Custom parsers are a great option when more flexibility is needed, or when assumptions can be made about the data format.

> this is a great piece of software

Glad to hear you are finding LogViewPlus helpful - thanks for your support! :)

Toby

RE: Help parsing

PIDtuner — Fri, 23 Jun 2023 08:15:52 GMT

Thank you for your answer,
I understand the preference for logs with the same format, but sadly that is not always the case. I have no freedom to modify these logs.
From your proposed options, the way I see it, and if I understand correctly:

1. Extracting File Name and Line columns with a message parser. This would do the trick, although subptimal to do two parsing steps.
2. Using Multi-pattern. This one is more optimal but looses a bit of information, since it skips the firts key value pair.

For me 2. is unacceptable since I need that bit of information. Then 1 would have to do.

Although, ideally (and I don't know how the parsing logic goes), and assuming the following:
- We know which is the start of line token [code]%d{dd/MM/yy-%H:mm:ss.ffffff}[/code]
- We know which is the end of line token [code]]%n[/code]
- We know where is themain content of the line [code]%m[/code]

This could be solved cleanly by taking %m as a pivot point for parsing and

- Parse all tokens at the left of %m from left to right.
- Parse all tokens at the right of %m form right to left.
- Once we parsed all tokens, %m is whatever remains.

Anyways, I'll get my license, since this is a great piece of software, wish it was cross platform though.

RE: Help parsing

LogViewPlus Support — Fri, 23 Jun 2023 07:15:09 GMT

Thanks for reaching out with this one. LogViewPlus parser configurations do have a bit of a learning curve and we are always happy to help.

LogViewPlus has a strong preference for log files where all log entries follow the same format. In this case, Log entry #3 is using a different pattern because it ends with two key value pairs rather than one.

A standard solution here would be to include the key value pairs as part of the message. Values could then be extracted later as part of a message parser:
[code]%d{dd/MM/yy-%H:mm:ss.ffffff} %t %p %m%n[/code]
However, in this case we can do a little better by using a multi-pattern which applies multiple parsing rules to the same log line on a 'first match' basis. For example:
[code]%d{dd/MM/yy-%H:mm:ss.ffffff} %t %p %m [%S] [%F:%L]%n
%d{dd/MM/yy-%H:mm:ss.ffffff} %t %p %m [%F:%L]%n[/code]

For log entries with two key value pairs, this will effectively skip the first key value pair and focus on the second line.

Hope that helps,

Toby