LogViewPlus Support » LogViewPlus Support » Configuration & Customization » PatternParser can't deal with date format

PatternParser can't deal with date format

mhkohne — Thu, 19 Sep 2019 11:40:07 GMT

I'm evaluating LogViewPlus and I've found something the pattern parser can't seem to deal with:
1565810719.699780: Starting encoder process NOW!
That line is one line of the output from a program called 'mencoder' (in this case version 1.3.0).
That number at the front is the # of seconds since Jan 1, 1970, but mencoder has decided to print it with fractional sections (instead of printing it as a whole number of microseconds or something like our other log files do).
Is there any way that I can get the pattern parser to deal with this properly? right now it just says parse error.
Alternately, can someone tell me what regex parser inputs look like? If I had some examples of regex parser definitions that included dates, I might be able to work it out myself (at least to the point of getting the seconds as the date and leaving the fraction as it's own column).

Thanks!

RE: PatternParser can't deal with date format

LogViewPlus Support — Thu, 19 Sep 2019 11:40:07 GMT

That's great - thanks for letting me know.

Toby

RE: PatternParser can't deal with date format

mhkohne — Thu, 19 Sep 2019 11:36:15 GMT

+x[quote]

[b]LogViewPlus Support - 9/14/2019[/b]
Hi,

The latest release of LogViewPlus (v2.3.25) includes a new 'ElapsedDecimal' built in date format. You can use this to parse your log files with:

[code]%d{ElapsedDecimal}: %m%n[/code]

Hope that helps. Please let me know if you have any further questions or issues.

Thanks,

Toby
[/quote]

Works great - thank you

RE: PatternParser can't deal with date format

LogViewPlus Support — Sat, 14 Sep 2019 06:19:20 GMT

Hi,

The latest release of LogViewPlus (v2.3.25) includes a new 'ElapsedDecimal' built in date format. You can use this to parse your log files with:

[code]%d{ElapsedDecimal}: %m%n[/code]

Hope that helps. Please let me know if you have any further questions or issues.

Thanks,

Toby

RE: PatternParser can't deal with date format

mhkohne — Tue, 20 Aug 2019 12:00:12 GMT

I'm perfectly happy with 'not pretty, but works fine' solutions.

I'll pre-process on my own for now - it only took me 10 minutes to write the shell script, and it works just fine for the moment.

Thank you again!

RE: PatternParser can't deal with date format

LogViewPlus Support — Tue, 20 Aug 2019 03:17:31 GMT

Thanks for the update and the additional sample log entries.

I think the likely solution here will be a new custom date format that can be used with the pattern parser. For example %d{TICK_WITH_FRACTIONAL_SECONDS}. It's not pretty, but parsing the log entry date is a core part of the application and we need a lot of flexibility.

If you are considering pre-processing your logs, you might also want to consider a custom parser:
https://www.logviewplus.com/docs/custom_parsers.html

You may also be able to use a post processor:
https://www.logviewplus.com/docs/post_processors.html

You can use a post processor or read the fractional "string" value and append this to the log entry date. This is a bit of a hack, but it may suffice until we can get a fix in.

Thanks again,

Toby

RE: PatternParser can't deal with date format

mhkohne — Mon, 19 Aug 2019 12:19:37 GMT

I agree it's an unusual date format, and I'm not the least bit surprised you've not seen it before, I certainly hadn't.
For the moment I may consider pre-processing the logs to turn those values into numbers of microseconds.

For reference, here's a couple of other lines:
1566086399.889997: openAviSegment took 0 ms
1566089999.981911: video buffer full - dropping frame
1566090000.14852: video buffer full - dropping frame
1566090000.47759: video buffer full - dropping frame
1566090000.80712: video buffer full - dropping frame
1566090000.113648: video buffer full - dropping frame
Note that as you'd expect for a fractional value, there's sometimes fewer digits after the decimal. I'm not clear whether this particular implementation can end up without the decimal at all (I've never noticed it hit exactly on the second before). I wouldn't be shocked to see the decimal entirely missing (1566086399), a single zero (1566086399.0) or even the entirely stupid just the decimal (1566086399. )

Also, for anyone coming along later, the link for the Pattern Parser docs in the reply above appears to have an extra invisible character at the end. Manually copy and paste it into your browser and delete the junk, and it's fine.

Thank you.

RE: PatternParser can't deal with date format

LogViewPlus Support — Sat, 17 Aug 2019 00:15:31 GMT

Hi,

Thanks for reporting this issue. I have been able to recreate the problem on my end and can confirm that this is a bug. This is an unusual date format that we have not seen before and we will get this issue fixed in the next release.

The best I can suggest for now is to use a multi-parse described here:
https://www.logviewplus.com/docs/pattern_parser.html

For example, I can parse this log entry to minute level accuracy with:
%r.%s: %m%n
%d: %m%n

The second parse format is used to handle the situation where no decimal place is found. This may not be necessary depending on your log file. Note that multi-parse configurations are not supported by the ParserWizard so you will need to enter the configuration manually.

Hope that helps. I will update you here when we have a fix available in beta.

Thanks,

Toby