PatternParser can't deal with date format


Author
Message
mhkohne
mhkohne
New Member (20 reputation)New Member (20 reputation)New Member (20 reputation)New Member (20 reputation)New Member (20 reputation)New Member (20 reputation)New Member (20 reputation)New Member (20 reputation)New Member (20 reputation)
Group: Forum Members
Posts: 6, Visits: 22
I'm evaluating LogViewPlus and I've found something the pattern parser can't seem to deal with:
1565810719.699780: Starting encoder process NOW!
That line is one line of the output from a program called 'mencoder' (in this case version 1.3.0).
That number at the front is the # of seconds since Jan 1, 1970, but mencoder has decided to print it with fractional sections (instead of printing it as a whole number of microseconds or something like our other log files do).
Is there any way that I can get the pattern parser to deal with this properly? right now it just says parse error.
Alternately, can someone tell me what regex parser inputs look like? If I had some examples of regex parser definitions that included dates, I might be able to work it out myself (at least to the point of getting the seconds as the date and leaving the fraction as it's own column).

Thanks!

LogViewPlus Support
LogViewPlus Support
Prestige User (3.9K reputation)Prestige User (3.9K reputation)Prestige User (3.9K reputation)Prestige User (3.9K reputation)Prestige User (3.9K reputation)Prestige User (3.9K reputation)Prestige User (3.9K reputation)Prestige User (3.9K reputation)Prestige User (3.9K reputation)
Group: Moderators
Posts: 1.1K, Visits: 3.7K
Hi,

Thanks for reporting this issue.  I have been able to recreate the problem on my end and can confirm that this is a bug.  This is an unusual date format that we have not seen before and we will get this issue fixed in the next release.​​

The best I can suggest​ for now is to use a multi-parse described here:
https://www.logviewplus.com/docs/pattern_parser.html​
​​
For example, I can parse this log entry to minute level accuracy with:
%r.%s: %m%n
%d: %m%n


The second parse format is used to handle the situation where no decimal place is found.  This may not be necessary depending on your log file.  Note that multi-parse configurations are not supported by the ParserWizard so you will need to enter the configuration manually.

​​
​Hope that helps.  I will update you here when we have a fix available in beta.

Thanks,

Toby​​​​​
​​​​​​​​
mhkohne
mhkohne
New Member (20 reputation)New Member (20 reputation)New Member (20 reputation)New Member (20 reputation)New Member (20 reputation)New Member (20 reputation)New Member (20 reputation)New Member (20 reputation)New Member (20 reputation)
Group: Forum Members
Posts: 6, Visits: 22
I agree it's an unusual date format, and I'm not the least bit surprised you've not seen it before, I certainly hadn't.
For the moment I may consider pre-processing the logs to turn those values into numbers of microseconds.

For reference, here's a couple of other lines:
1566086399.889997: openAviSegment took 0 ms
1566089999.981911: video buffer full - dropping frame
1566090000.14852: video buffer full - dropping frame
1566090000.47759: video buffer full - dropping frame
1566090000.80712: video buffer full - dropping frame
1566090000.113648: video buffer full - dropping frame
Note that as you'd expect for a fractional value, there's sometimes fewer digits after the decimal. I'm not clear whether this particular implementation can end up without the decimal at all (I've never noticed it hit exactly on the second before). I wouldn't be shocked to see the decimal entirely missing (1566086399), a single zero (1566086399.0) or even the entirely stupid just the decimal (1566086399. )

Also, for anyone coming along later, the link for the Pattern Parser docs in the reply above appears to have an extra invisible character at the end. Manually copy and paste it into your browser and delete the junk, and it's fine.

Thank you.


LogViewPlus Support
LogViewPlus Support
Prestige User (3.9K reputation)Prestige User (3.9K reputation)Prestige User (3.9K reputation)Prestige User (3.9K reputation)Prestige User (3.9K reputation)Prestige User (3.9K reputation)Prestige User (3.9K reputation)Prestige User (3.9K reputation)Prestige User (3.9K reputation)
Group: Moderators
Posts: 1.1K, Visits: 3.7K
Thanks for the update and the additional sample log entries. 

I think the likely solution here will be a new custom date format that can be used with the pattern parser.  For example %d{TICK_WITH_FRACTIONAL_SECONDS}.  It's not pretty, but parsing the log entry date is a core part of the application and we need a lot of flexibility.

If you are considering pre-processing your logs, you might also want to consider a custom parser:
https://www.logviewplus.com/docs/custom_parsers.html

You may also be able to use a post processor:
https://www.logviewplus.com/docs/post_processors.html

You can use a post processor or read the fractional "string" value and append this to the log entry date.​  This is a bit of a hack, but it may suffice until we can get a fix in.

Thanks again,

Toby​​​​​​​​​​​​​​​
mhkohne
mhkohne
New Member (20 reputation)New Member (20 reputation)New Member (20 reputation)New Member (20 reputation)New Member (20 reputation)New Member (20 reputation)New Member (20 reputation)New Member (20 reputation)New Member (20 reputation)
Group: Forum Members
Posts: 6, Visits: 22
I'm perfectly happy with 'not pretty, but works fine' solutions.

I'll pre-process on my own for now - it only took me 10 minutes to write the shell script, and it works just fine for the moment.

Thank you again!

LogViewPlus Support
LogViewPlus Support
Prestige User (3.9K reputation)Prestige User (3.9K reputation)Prestige User (3.9K reputation)Prestige User (3.9K reputation)Prestige User (3.9K reputation)Prestige User (3.9K reputation)Prestige User (3.9K reputation)Prestige User (3.9K reputation)Prestige User (3.9K reputation)
Group: Moderators
Posts: 1.1K, Visits: 3.7K
Hi,

The latest release of LogViewPlus (v2.3.25) includes a new 'ElapsedDecimal' built in date format.  You can use this to parse your log files with:

%d{ElapsedDecimal}: %m%n


Hope that helps.  Please let me know if you have any further questions or issues.

Thanks,

Toby
Edited 5 Years Ago by LogViewPlus Support
mhkohne
mhkohne
New Member (20 reputation)New Member (20 reputation)New Member (20 reputation)New Member (20 reputation)New Member (20 reputation)New Member (20 reputation)New Member (20 reputation)New Member (20 reputation)New Member (20 reputation)
Group: Forum Members
Posts: 6, Visits: 22
LogViewPlus Support - 9/14/2019
Hi,

The latest release of LogViewPlus (v2.3.25) includes a new 'ElapsedDecimal' built in date format.  You can use this to parse your log files with:

%d{ElapsedDecimal}: %m%n


Hope that helps.  Please let me know if you have any further questions or issues.

Thanks,

Toby


Works great - thank you

LogViewPlus Support
LogViewPlus Support
Prestige User (3.9K reputation)Prestige User (3.9K reputation)Prestige User (3.9K reputation)Prestige User (3.9K reputation)Prestige User (3.9K reputation)Prestige User (3.9K reputation)Prestige User (3.9K reputation)Prestige User (3.9K reputation)Prestige User (3.9K reputation)
Group: Moderators
Posts: 1.1K, Visits: 3.7K
That's great - thanks for letting me know.

Toby
GO

Merge Selected

Merge into selected topic...



Merge into merge target...



Merge into a specific topic ID...




Similar Topics

Login

Explore
Messages
Mentions
Search