LogViewPlus Support » LogViewPlus Support » Features & Suggestions » Add ability to parse logfmt logs

Add ability to parse logfmt logs

ethirolle — Fri, 18 Dec 2020 16:49:35 GMT

Apps like Grafana use a log format that is sometimes called "logfmt", and which looks like this:

t=2020-12-18T16:27:55+0000 lvl=info msg="Request Completed" logger=context userId=2 orgId=1 uname=erict method=GET path=/api/usage/dashboard/14/views/recent status=200 remote_addr=24.168.218.173 time_ms=77 size=2 referer="https://ericgce.grafana.net/d/HvRjRI5Gz/windows-node-eric-laptop?orgId=1&refresh=5m" traceID=409dae83dc12a6a7

I.e., it is structured as a space-delimited list of key=value pairs. The tricky part of parsing this log format is: each logger may have a different set of keys (and there are many different loggers populating log messages into this one log file). So if you want to parse this type of log into a table format, I think you would first need to read in the whole file, find all the possible keys, and make a column for each.

I am doing a lot of Grafana log analysis lately, and have not been able to use LogViewPlus for any of it. I really miss my LogViewPlus! Lol

Do you think it would be possible to add a logfmt parser to LogViewPlus?

Thanks!!
Eric

RE: Add ability to parse logfmt logs

LogViewPlus Support — Fri, 18 Dec 2020 16:49:35 GMT

Hi Eric,

Thanks for the feedback.

We can definitely parse this type of file. The end result would be something like the JSON Parser where you need to define the fields you are interested in prior to opening the file.

Rather than create a "Logfmt Parser", what I would like to do is create a generic "Key Value Pair Parser". The generic version could parse the log entry above, but also potentially extract data from XML or JSON formats. The purpose here wouldn't be to replace those parsers, but just to have a more generic parser that could be used to parse some of these lesser known log formats.

I think this is something that would really improve LogViewPlus. The problem is scheduling. We have a bit of a backlog at the moment, but hopefully we can have something for you before July. We will prioritize this, but there are a few other big features we need to get out the door first.

I will post back here when I have a BETA version for you.

Thanks again!

Toby