Help parsing


Author
Message
PIDtuner
PIDtuner
New Member (9 reputation)New Member (9 reputation)New Member (9 reputation)New Member (9 reputation)New Member (9 reputation)New Member (9 reputation)New Member (9 reputation)New Member (9 reputation)New Member (9 reputation)
Group: Forum Members
Posts: 2, Visits: 3
If I can get around this, I am definetively getting my license.
So I have an unfortunate log with entries as follow:
22/06/23-09:49:58.007050 Thread INFO  read 4 bytes from sock 234. Pending: 0 errno=0 [Comms.cpp:517]
22/06/23-09:49:58.007064 Thread INFO Read from server: RpcGlobalCallback. con: sock=234 [Rpc.cpp:1060]
22/06/23-09:49:58.007072 Thread INFO Prepare for read 4 bytes: [sock=234] [Comms.cpp:359]
22/06/23-09:49:58.007102 Thread INFO No more action Connection::handleConnectionEvent sock = 234. mState = 1 [Comms.cpp:590]
22/06/23-09:49:58.007108 Thread INFO Epolled: 9 flags = EPOLLIN [Comms.cpp:192]

I am using the following pattern:
%d{dd/MM/yy-%H:mm:ss.ffffff} %t %p %m [%F:%L]%n

But I keep getting bad lines like the highlighted one below, where the File Name column is wrong:

Tags
LogViewPlus Support
LogViewPlus Support
Supreme Being (5.3K reputation)Supreme Being (5.3K reputation)Supreme Being (5.3K reputation)Supreme Being (5.3K reputation)Supreme Being (5.3K reputation)Supreme Being (5.3K reputation)Supreme Being (5.3K reputation)Supreme Being (5.3K reputation)Supreme Being (5.3K reputation)
Group: Moderators
Posts: 1.1K, Visits: 3.7K
Thanks for reaching out with this one.  LogViewPlus parser configurations do have a bit of a learning curve and we are always happy to help.

LogViewPlus has a strong preference for log files where all log entries follow the same format.  In this case, Log entry #3 is using a different pattern because it ends with two key value pairs rather than one.

A standard solution here would be to include the key value pairs as part of the message.  Values could then be extracted later as part of a message parser:
%d{dd/MM/yy-%H:mm:ss.ffffff} %t %p %m%n

However, in this case we can do a little better by using a multi-pattern which applies multiple parsing rules to the same log line on a 'first match' basis.  For example:
%d{dd/MM/yy-%H:mm:ss.ffffff} %t %p %m [%S] [%F:%L]%n
%d{dd/MM/yy-%H:mm:ss.ffffff} %t %p %m [%F:%L]%n



For log entries with two key value pairs, this will effectively skip the first key value pair and focus on the second line.



Hope that helps,

Toby
Edited 10 Months Ago by LogViewPlus Support
PIDtuner
PIDtuner
New Member (9 reputation)New Member (9 reputation)New Member (9 reputation)New Member (9 reputation)New Member (9 reputation)New Member (9 reputation)New Member (9 reputation)New Member (9 reputation)New Member (9 reputation)
Group: Forum Members
Posts: 2, Visits: 3
Thank you for your answer,
I understand the preference for logs with the same format, but sadly that is not always the case. I have no freedom to modify these logs.
From your proposed options, the way I see it, and if I understand correctly:

1. Extracting File Name and Line columns with a message parser. This would do the trick, although subptimal to do two parsing steps.
2. Using Multi-pattern. This one is more optimal but looses a bit of information, since it skips the firts key value pair.

For me 2. is unacceptable since I need that bit of information. Then 1 would have to do.

Although, ideally (and I don't know how the parsing logic goes), and assuming the following:
- We know which is the start of line token
%d{dd/MM/yy-%H:mm:ss.ffffff}

- We know which is the end of line token 
]%n

- We know where is themain content of the line 
%m


This could be solved cleanly by taking %m as a pivot point for parsing and

- Parse all tokens at the left of %m from left to right.
- Parse all tokens at the right of %m form right to left.
- Once we parsed all tokens, %m is whatever remains.

Anyways, I'll get my license, since this is a great piece of software, wish it was cross platform though.
LogViewPlus Support
LogViewPlus Support
Supreme Being (5.3K reputation)Supreme Being (5.3K reputation)Supreme Being (5.3K reputation)Supreme Being (5.3K reputation)Supreme Being (5.3K reputation)Supreme Being (5.3K reputation)Supreme Being (5.3K reputation)Supreme Being (5.3K reputation)Supreme Being (5.3K reputation)
Group: Moderators
Posts: 1.1K, Visits: 3.7K
If you need the additional information, you can parse it out with something like;
%d{dd/MM/yy-%H:mm:ss.ffffff} %t %p %m [%s{Key}=%s{Value}] [%F:%L]%n

The specifier [%s{Key}=%s{Value}] will extract the information into new Key and Value columns.  I left this out earlier because it wasn't clear the information was needed and it makes the pattern a little more complicated.  Sorry for the confusion.

We have spent of lot of time optimizing the LogViewPlus pattern parser for performance and flexibility.  The model we have decided on for this parser is forward-only, but you are absolutely right that other approaches may make more sense in certain scenarios.  Custom parsers are a great option when more flexibility is needed, or when assumptions can be made about the data format.

> this is a great piece of software

Glad to hear you are finding LogViewPlus helpful - thanks for your support!  Smile

Toby

GO

Merge Selected

Merge into selected topic...



Merge into merge target...



Merge into a specific topic ID...




Similar Topics

Login

Explore
Messages
Mentions
Search