LogViewPlus Support » LogViewPlus Support » Help & Support » Another: scrambled fields with merged logs

Another: scrambled fields with merged logs

Vankog — Mon, 31 Mar 2025 17:43:04 GMT

Hello,
after https://www.logviewplus.com/forum/2294/Scrambled-fields-when-parsing-json-properties-with-n I am now on Beta 3.1.18.
But the previous version(s) already had this issue. It is just extremely tedious to report it properly and prepare the examples. So I hope this effort leads to fruition.

Given at least two distinct log types with some overlap in fields:
1. web-access log, like attached web.log(.txt)
with this DsvParser pattern:
[code]%d{Elapsed}|%S{remote address}|%S{remote user}|%S{host name}|%S{Port}|%S{runtime}|%S{path}|%S{user agent}|%S{cookie}|%S{referrer}|%S{session}|%S{request}|%S{response source server/cache}|%S{response status}|%S{receive time}|%S{wait time}|%S{processing time}|%S{resolve time}|%S{transfer time}|%S{request method}|%S{PGID}|%S{PGID state}|%S{personalized flag}|%S{request type}|%S{response size external:internal}|%S{transfer size}|%S{secure flag}|%S{robot flag}|%S{binary flag}|%S{response status}|%S{page name}%n[/code]

2. app.log, like the attached app.log(.txt)
with PatternParser pattern:
[code][%d{yyyy-MM-dd %H:mm:ss.fff zzzz}] %-5p %s{host name} %s{installation} %s{server name} [%a] [%S{application}] %c [%marker] [%S{request type}] [%S{session}] [%S{request}] "%t" %m%n[/code]

When these are merged, the columns get unioned. However, columns that don't exist in either of the two logs types, have a chance to be filled with phantom values from another column, making analysis quite hard.

Example:
I sorted the columns alphabetically to make the analysis easier.
web log has these fields:

(see attached web log.png)

app log has these fields:

(see attached app log.png)

When merged, we get this mess:

(see attached merged log.png)

This also pollutes the data of other tools, like the Graph Reports:

(see attached polluted user agent.png)

I really hope this helps to finally fix this annoying issue.

Thanks and best regards,
Daniel

BTW: During my efforts to gather this data, I ran into just two other issues:
1. After I sorted all the columns in the app- and web-logs alphabetically and adjusted their width, the merged log refused to union the columns. All I got, were some common columns. If I remember correctly from the app logs, but I think not all of them.
Only after I reset the columns of app and web log to defaults, the union happened again.
However, with random changes of the column order and width, I was not able to reproduce this issue. I had not enough motivation to reorder all the columns completely, just for this.

2. Something minor: I have noticed, that the web log maps a field "%d{Elapsed}" to map the timestamp properly. As this also seems to be a built-in and calculated field of LogViewPlus, this probably causes some inconsistent behavior as you can see in the screenshots. (my guess)

RE: Another: scrambled fields with merged logs

LogViewPlus Support — Mon, 31 Mar 2025 17:43:04 GMT

Hi Daniel,

I just wanted to let you know that we have now released LogViewPlus v3.1.20 as a BETA release

This release addresses the merge column issues discussed above. The revised view should looks something like:

Hope that helps. Thanks again for bringing this issue to our attention.

Toby

RE: Another: scrambled fields with merged logs

LogViewPlus Support — Wed, 22 Jan 2025 20:08:12 GMT

Hi Daniel,

Thanks for reporting this issue. This is a brilliant bug report - thanks for taking the time!

I have been able to recreate the issue. I have not yet tracked down a root cause, but this is a serious issue for us. We will definately have a fix for the next release and we will also bring it forward as much as we can.

I will post back here as soon as we have a fix.

Regarding the sub-issues you highlighted.

1. It sounds like there was some internal data corruption with how columns are being tracked. This could be related to the root issue. I will take a closer look into this as we investigate the main issue as well as the Save column settings for merged files issue. Hopefully we can identify a cause and get something into the next release.

2. The %d{Elapsed} specifier should produce a consistent timestamp. "Elapsed" is a generic implementation of Elapsed Date Times where LogViewPlus uses heuristics to determine how the timestamp should be handled. It is actually missing from that documentation page, so I will update that as well.

Thanks again!

Toby