LogViewPlus Support » LogViewPlus Support » Features & Suggestions » Deduplicate log entries

Deduplicate log entries

SimonD — Fri, 16 Apr 2021 13:47:45 GMT

Hi Toby,

I'd like to be able to detect and hide consecutive identical log entries (or sequences of log entries), while adding the information how many times the entry or sequence of entries was repeated.

I achieved the duplicate detection and filtering through a custom filter implementation, but couldn't find a good way to add the number of repetitions anywhere.
Keeping a reference to the first LogEntry instance of the duplication series and prepending the number of duplications to the message worked as long as I was in the filter, but the modified message persisted even after removing the filter and generally this doesn't seem like a clean solution.

Is there some better way to achieve this, and if not, could you add it to a future version?

Thanks,
Simon

RE: Deduplicate log entries

LogViewPlus Support — Fri, 16 Apr 2021 13:47:45 GMT

No problem Simon. Hope that helps,

Toby

RE: Deduplicate log entries

SimonD — Fri, 16 Apr 2021 13:26:44 GMT

I actually have a custom parser for these logs already, but haven't considered using it for the deduplication yet.
I'll give it a try, thanks!

RE: Deduplicate log entries

LogViewPlus Support — Fri, 16 Apr 2021 11:43:49 GMT

Thanks for the screenshots Simon.

The problem makes sense, but I think this is a niche issue. LogViewPlus currently does not have the ability to 'hide' a log entry.

I think your customer filter approach is pretty good. Have you considered writing a custom parser? This would give you a lot more control including the ability to remove / count log entries. You could even add an additional column to track the number of duplications.

Toby

RE: Deduplicate log entries

SimonD — Fri, 16 Apr 2021 11:04:18 GMT

Thanks for the quick response!
Didn't know about the message template column, very cool feature.

However my goal is more about filtering out consecutive duplicates to reduce clutter and improve readability of the other log entries. Basically my log contains a lot of breadcrumbs for various entities, which I use to track their behavior over time (usually filtering for a specific entity). As part of their normal behavior, an entity will often repeat the same breadcrumb as they repeat an action, which can clutter the log view. So I want to filter consecutive duplicate breadcrumbs to have a better o, while still seeing that they happened (hence the need for a duplicate count).

I guess I'll just keep using my hacky custom filter for now, since it works well enough:
original log:

deduplicated log:

Thanks,
Simon

RE: Deduplicate log entries

LogViewPlus Support — Fri, 16 Apr 2021 09:46:13 GMT

Hi Simon,

Thanks for the feedback.

Newer versions of LogViewPlus have a hidden column called 'Message Template'. This column attempts to identify messages by their structure. If you show this column and group by it, you should have a report on the number of times the message is used. This might help a bit.

I would be very interested on any feedback with message templating but, generally, I think removing and tracking duplicates may a niche requirement and I am not sure how to work it into the product.

I have one other suggestion. What happens if you invert your custom filter? This should show all items excluded by the filter - which would give you an idea of the number of duplicates.

Hope that helps,

Toby