Add ability to parse logfmt logs


Author
Message
ethirolle
ethirolle
New Member (7 reputation)New Member (7 reputation)New Member (7 reputation)New Member (7 reputation)New Member (7 reputation)New Member (7 reputation)New Member (7 reputation)New Member (7 reputation)New Member (7 reputation)
Group: Forum Members
Posts: 1, Visits: 3
Apps like Grafana use a log format that is sometimes called "logfmt", and which looks like this:

t=2020-12-18T16:27:55+0000 lvl=info msg="Request Completed" logger=context userId=2 orgId=1 uname=erict method=GET path=/api/usage/dashboard/14/views/recent status=200 remote_addr=24.168.218.173 time_ms=77 size=2 referer="https://ericgce.grafana.net/d/HvRjRI5Gz/windows-node-eric-laptop?orgId=1&refresh=5m" traceID=409dae83dc12a6a7

I.e., it is structured as a space-delimited list of key=value pairs. The tricky part of parsing this log format is: each logger may have a different set of keys (and there are many different loggers populating log messages into this one log file). So if you want to parse this type of log into a table format, I think you would first need to read in the whole file, find all the possible keys, and make a column for each.

I am doing a lot of Grafana log analysis lately, and have not been able to use LogViewPlus for any of it. I really miss my LogViewPlus! Lol

Do you think it would be possible to add a logfmt parser to LogViewPlus?

Thanks!!
Eric
LogViewPlus Support
LogViewPlus Support
Supreme Being (5.3K reputation)Supreme Being (5.3K reputation)Supreme Being (5.3K reputation)Supreme Being (5.3K reputation)Supreme Being (5.3K reputation)Supreme Being (5.3K reputation)Supreme Being (5.3K reputation)Supreme Being (5.3K reputation)Supreme Being (5.3K reputation)
Group: Moderators
Posts: 1.1K, Visits: 3.7K
Hi Eric,

Thanks for the feedback.

We can definitely parse this type of file.  The end result would be something like the JSON Parser where you need to define the fields you are interested in prior to opening the file.

Rather than create a "Logfmt Parser", what I would like to do is create a generic "Key Value Pair Parser".  The generic version could parse the log entry above, but also potentially extract data from XML or JSON formats.  The purpose here wouldn't be to replace those parsers, but just to have a more generic parser that could be used to parse some of these lesser known log formats.

I think this is something that would really improve LogViewPlus.  The problem is scheduling.  We have a bit of a backlog at the moment, but hopefully we can have something for you before July.  We will prioritize this, but there are a few other big features we need to get out the door first.

I will post back here when I have a BETA version for you.

Thanks again!

Toby
GO

Merge Selected

Merge into selected topic...



Merge into merge target...



Merge into a specific topic ID...




Similar Topics

Login

Explore
Messages
Mentions
Search