Is it possible to ignore invalid lines in json log


Author
Message
petr
petr
Junior Member (82 reputation)Junior Member (82 reputation)Junior Member (82 reputation)Junior Member (82 reputation)Junior Member (82 reputation)Junior Member (82 reputation)Junior Member (82 reputation)Junior Member (82 reputation)Junior Member (82 reputation)
Group: Forum Members
Posts: 7, Visits: 19
I've been struggling with this issue for some time.
Some of the lines in our log file (SeriLog json) are invalid (cutoff by syslog)
Every time I load up the file to LogView it crashes (standard error reporter window).

Is there a way to make logivew skip the lines? Even better would be to display them as text but that's not necessary.

Thanks
Replies
LogViewPlus Support
LogViewPlus Support
Supreme Being (12K reputation)Supreme Being (12K reputation)Supreme Being (12K reputation)Supreme Being (12K reputation)Supreme Being (12K reputation)Supreme Being (12K reputation)Supreme Being (12K reputation)Supreme Being (12K reputation)Supreme Being (12K reputation)
Group: Moderators
Posts: 1.3K, Visits: 4.6K
Thanks for the sample log entry.

I was not able to recreate the 'crash' on my end.  If you have not done so already, can you please click the 'Yes' button on the dialog above?  This will crash the application and send us an exception report.  Often we can solve these kinds of issues by looking at the exception details.

It makes sense that the file is not parsing because the log entry is invalid.  The invalid JSON actually confuses LogViewPlus and makes it difficult to know where the log entry ends and the next one begins.   Note that other log files may have multi-line JSON and the parser needs to be able to handle this scenario too.

The only work around I can think of is to have a setting which tells the parser "I expect all JSON to be on a single log line".  However, this setting would be confusing for a lot of users.  I would like to be able to handle this scenario, but I am short on ideas on how to do so.

You might want to look at writing a custom parser.  The fact that the JSON is all on one line would make the implementation pretty easy.  A custom parser would allow you flexibility in handling exceptions.

What I would suggest is to use the PatternParser instead.  I can parse this log entry with:
{"Timestamp":"%d{yyyy-MM-ddT%H:mm:ss.fffffffzzzz}","Level":"%p","MessageTemplate":"%S{MessageTemplate}","%S":"%m%n


Note that the configuration above ignores the name of the final JSON tag.  It simply takes the last element as the log entry message and does not look for a close.

The downside of this approach is that the log entries will not be pretty printed as JSON, you will need to change the pretty print style manually:


Hope that makes sense.

Toby
petr
petr
Junior Member (82 reputation)Junior Member (82 reputation)Junior Member (82 reputation)Junior Member (82 reputation)Junior Member (82 reputation)Junior Member (82 reputation)Junior Member (82 reputation)Junior Member (82 reputation)Junior Member (82 reputation)
Group: Forum Members
Posts: 7, Visits: 19
Thank you for the explanation and advice. I didn't realize there were logs with multiline json. You suggestion makes sense to me, it can be generalized to multi-line vs single-line parsing and it could be useful. Although I've already solved my issue by finally reconfiguring rsyslog on prod (rsyslog config is the real pain point here not logviewplus Smile ).

I'd just sent the crash report (2021-03-12 09:30 UTC+1)




LogViewPlus Support
LogViewPlus Support
Supreme Being (12K reputation)Supreme Being (12K reputation)Supreme Being (12K reputation)Supreme Being (12K reputation)Supreme Being (12K reputation)Supreme Being (12K reputation)Supreme Being (12K reputation)Supreme Being (12K reputation)Supreme Being (12K reputation)
Group: Moderators
Posts: 1.3K, Visits: 4.6K
Glad that helped.  Thanks for the crash report - I will take a look.

Yes - the rsyslog configuration is the real pain point.  It's good you were able to make this change, but I appreciate that this is not always possible for all users.  I will keep thinking about this problem and see what I can come up with.

Thanks for highlighting this issue!

Toby
GO

Merge Selected

Merge into selected topic...



Merge into merge target...



Merge into a specific topic ID...





Similar Topics

Login

Explore
Messages
Mentions
Search