Hello,

I often have to work with very large logs - in the realm of 1 million records and easily more, gigabytes in log file size.

I usually merge these files and then start drilling down by using filters, like level filters and text filters (sometimes I also combine them with the boolean filter logic):

(see attached example.png)

To my surprise, when I add nested filters, the search time still takes quite a while (several seconds), even though the outer filter only has a few entries as result (in the realm of ~1-100 entries).

My guess is that the inner filters still run against the whole data set and then the result is merged with the outer filters.

My expectation, though, is that the inner filters should run against the already filtered outer results, in theory leading to much shorter runtimes.

Is there something I or you can do to improve the performance of nested filters?

Currently, I am on Beta 3.1.18, but this was already an issue with older versions.

Thanks and best regards,
Daniel

Hi Daniel,

Filters are processed from the root log file out such that children are processed after parents.  So in the screenshot above the search "GetEntryFromMap" would occur before the search "ProcessMyMachine" would occur before the search "ViewProduct".

The best thing to do in this case is for us to run LogViewPlus in an application profilier while we recreate the issue.  This may help to pinpoint a particular issue.  If you are interested in exploring this option, please feel free to contact us and we can discuss.  We would probably need to transfer data over email.

That said, we have already performance tested filtering and it is not guarenteed we will be able to further improve application performance.  The issue here is likely to be related to the volume of data being processed.

The next best thing to do is to rewrite your filters to process from parent to child.  Ideally, parent filters would work with smaller data sets (columns which contain less data, such as a Logger column or the log Level).  Fitlering out rows early should result in less work for the child.

Hope that helps,

Toby

The next best thing to do is to rewrite your filters to process from parent to child. Ideally, parent filters would work with smaller data sets (columns which contain less data, such as a Logger column or the log Level). Filtering out rows early should result in less work for the child.

That's exactly what I'm doing. 
In the example, the merge node has about 1.2 mio. records as you can see.
Actually, now 1.3 mio.:
"GetEntryFromMap" case-sensitive in "Message" has 49 results and took about 14 seconds.
exclude "ProcessMyMachines" case-sensitive in "Message" has 36 results left and took abouit 15 seconds
exclude "ViewProduct-RenderProductComponents" case-sensitive in "Message" has no results left (meaning, I found all cases) and took about 17 seconds.
It feels like all filters where re-executed every time or something.

Inverting the exclude to includes has not much impact. It still took "ProcessMyMachines" 14 seconds.

I am happy to assist in profiling the app. Shoot me a mail and I will see how I can help.

Thanks for the update. 

That's not the behaviour I would expect.  This doesn't sound like an issue that would be specific to your log files, so I can start by looking at the log files I have available.

To start a off-line dialog, please use the contact us page and provide your email address.