Extend Custom Filters API


Author
Message
AndreasP
AndreasP
New Member (23 reputation)New Member (23 reputation)New Member (23 reputation)New Member (23 reputation)New Member (23 reputation)New Member (23 reputation)New Member (23 reputation)New Member (23 reputation)New Member (23 reputation)
Group: Forum Members
Posts: 19, Visits: 120
Dear Toby,

when analyzing complex log files or problems I sometimes would like to be able to make very special evaluations like the following.

For example I have a list of 50 log files and I want to see the first 10 and last 10 log entries of every log file.

Or another example is, that our log files contain a list of hardware components in our device including serial numbers. And for a list of 50 consecutive log files I would like to see if and where hardware changes between log files have occurred.

Both cases I think could be realized by a custom filter on a merged log file list. However the current ILogFilter interface only allows to see a single log file entry. For the above cases I would have to also check other, neighboring, log file entries (to find the position of the log entries in the log files or the find the hardware components list of the preceding log file).

Do you think such an extension would be feasible?

Kind Regards
Andreas
LogViewPlus Support
LogViewPlus Support
Gaining Respect (162 reputation)Gaining Respect (162 reputation)Gaining Respect (162 reputation)Gaining Respect (162 reputation)Gaining Respect (162 reputation)Gaining Respect (162 reputation)Gaining Respect (162 reputation)Gaining Respect (162 reputation)Gaining Respect (162 reputation)
Group: Moderators
Posts: 129, Visits: 505
That's really interesting Andreas.

My first thought is that we could achieve this by adding a Next and Previous properties on the LogEntry object - converting it into a item in a linked list.  Unfortunately, this won't work in your scenario as the Next and Previous would be relevant to the origin log file - not the merged log file.

So, I think what is needed here is a GetNext() and GetPrevious() delegate.  These could then be wired up such that when called they will retrieve the next or previous LogEntry in the current view.  So, the next / previous calls will return different log entries depending on how the view is filtered or merged.  I think this is a relatively easy change that would unlock lots of interesting scenarios.

Let me know if you think this will meet your requirements and I will see if we can get it into the next release.

Thanks!

Toby
AndreasP
AndreasP
New Member (23 reputation)New Member (23 reputation)New Member (23 reputation)New Member (23 reputation)New Member (23 reputation)New Member (23 reputation)New Member (23 reputation)New Member (23 reputation)New Member (23 reputation)
Group: Forum Members
Posts: 19, Visits: 120
Hi Toby,

sounds like a good idea. From what I can see from the LogEntry class interface I should have all information I need together with the GetNext and GetPrevious methods.

Andreas
LogViewPlus Support
LogViewPlus Support
Gaining Respect (162 reputation)Gaining Respect (162 reputation)Gaining Respect (162 reputation)Gaining Respect (162 reputation)Gaining Respect (162 reputation)Gaining Respect (162 reputation)Gaining Respect (162 reputation)Gaining Respect (162 reputation)Gaining Respect (162 reputation)
Group: Moderators
Posts: 129, Visits: 505
Hi Andreas,

This API has now been implemented in the latest BETA release (v2.3.6).  The documentation on the API is still pending, but I created a sample project which you can download here: https://www.logviewplus.com/docs/running_the_samples.html  Please see the "CustomAnalyzer" project.

I had to implement the API a little bit differently than discussed above as the Next / Previous methods actually return an IEnumerable.  This is for performance assuming that some users will want to continue finding the "next / previous" entries.

Thanks for the feedback!  Please let me know if you have any further questions or issues.

Toby
AndreasP
AndreasP
New Member (23 reputation)New Member (23 reputation)New Member (23 reputation)New Member (23 reputation)New Member (23 reputation)New Member (23 reputation)New Member (23 reputation)New Member (23 reputation)New Member (23 reputation)
Group: Forum Members
Posts: 19, Visits: 120
Hi Toby,

will this IEnumerable interface also be available to the "CustomFilter" plugins?
With the "CustomAnalyzer" plugin I can basically do the required filtering, but I can't view the filtered log entries in the Log Entry Grid. Or what is even more interesting is to combine of further filter the results with other "normal" filters.

I think it basically would be another kind of Show method like
public bool[] Show(IReadOnlyList<LogEntry> logEntries)
that processes all logEntries at once instead of calling the current Show method for each entry.

What do you think?

Regards
Andreas
LogViewPlus Support
LogViewPlus Support
Gaining Respect (162 reputation)Gaining Respect (162 reputation)Gaining Respect (162 reputation)Gaining Respect (162 reputation)Gaining Respect (162 reputation)Gaining Respect (162 reputation)Gaining Respect (162 reputation)Gaining Respect (162 reputation)Gaining Respect (162 reputation)
Group: Moderators
Posts: 129, Visits: 505
The Previous / Next methods are off of the LogEntry - so there shouldn't be any problem with calling these methods from a custom filter using the existing interface.

However.  It just occurred to me that the methods are working off of the current filter.  In the case of a custom filter, they will be reading the previous entry in the filter.  Next() will always return null because you haven't create a 'next' entry yet!  :-)  Not very helpful.

I think we will need to revisit these methods to accept an enum parameter:
LookupSource
{
LogFile,
CurrentFilter,
ParentFilter
}


Then, in your custom filter, you could call entry.Next(LookupSource.Parent) to get the next log entry from the parent filter.

The initial call into Next or Previous basically does a scan of the view to find the entry and uses this as the starting point for the Enumerable.  So, there may be a performance hit if a large filter were to scan each entry.  For now, I would like to cross this bridge when we come to it.  For example, it may be better to do a date search rather than a scan.  I am trying to keep the API simple and I don't want to create a IFilter2 interface that does almost the same thing unless a very clear need exists.

Also, keep in mind that filters need to work in tail mode.  The method above would probably need to refresh the filter on every tail tick - which would not be ideal.

Hope that makes sense.  I will get a new version of the API out before the release is moved out of BETA.  I should have something for you in the next few days.

Thanks,

Toby
Edited 2 Months Ago by LogViewPlus Support
AndreasP
AndreasP
New Member (23 reputation)New Member (23 reputation)New Member (23 reputation)New Member (23 reputation)New Member (23 reputation)New Member (23 reputation)New Member (23 reputation)New Member (23 reputation)New Member (23 reputation)
Group: Forum Members
Posts: 19, Visits: 120
Hi Toby,

I think I completely misunderstood your post about the beta version. After your valuable additional explanations, some digging into c# (I'm not really an c# expert) and looking on the example again, I now understand.

Thanks and looking forward to the next beta.

Andreas
LogViewPlus Support
LogViewPlus Support
Gaining Respect (162 reputation)Gaining Respect (162 reputation)Gaining Respect (162 reputation)Gaining Respect (162 reputation)Gaining Respect (162 reputation)Gaining Respect (162 reputation)Gaining Respect (162 reputation)Gaining Respect (162 reputation)Gaining Respect (162 reputation)
Group: Moderators
Posts: 129, Visits: 505
Hi Andreas,

The latest LogViewPlus BETA v2.3.7 has updated the API as discussed above.  I have also updated the sample code project here: https://www.logviewplus.com/docs/running_the_samples.html

The new enumeration is slightly different than discussed.  The "LogFile" value, needs to be either:
ViewLogFile - The root log file of the current filter tree.
SourceLogFile - The log file which originally contained the log entry.

Most of the time, these will be the same.  The distinction is necessary to allow you to search in the original log file instead of (possibly) a merged log file.  If that makes sense?

Have a look and let me know what you think.  Please don't hesitate to ask if you get stuck or something is not clear.  Using a 3rd party API is never easy and it is even worse when it is new / undocumented / un-googleable.  :-)

Thanks,

Toby
AndreasP
AndreasP
New Member (23 reputation)New Member (23 reputation)New Member (23 reputation)New Member (23 reputation)New Member (23 reputation)New Member (23 reputation)New Member (23 reputation)New Member (23 reputation)New Member (23 reputation)
Group: Forum Members
Posts: 19, Visits: 120
Hi Toby,

I implemented a first filter and tested the new API. It took me some time to figure out the use of the new functions, but at the end I got it working. And basically it works. However I experienced quite some performance problems when working on real world cases.

I have merged 35 log files, that have about 1.7Mio records in total (typical scenario). I filtered out the relevant log entries with a normal "Logger Filter" to about 8000 entries, where I apply my new filter. It takes about several minutes to execute. If I prepare a single log file to contain only the relevant 8000 log file entries (ok, not exactly the same, but equivalent), the custom filter runs very quickly as expected.

So it seems that even I apply the custom filter only to the filtered view of 8000 entries, the parent view of 1.7 Mio entries somehow extremely slows down the filter. Till now I was not able to find out what is really taking so long. Here is code:

public bool Show(LogEntry logEntry)
{
if (logEntry.Logger.Contains("ProcessingEngineManager::logBoardEepromInfos"))
{
String boardSpec;
String message = logEntry.Message;
var match = Regex.Match(message, @"^(theBoard=\w+, iInstance=\d+, ID=\d+), MatNo");
if (match.Success)
{
boardSpec = match.Groups[1].Value;

foreach (LogEntry entry in logEntry.FindPrevious(LookupSource.ParentFilter))
{
// search for the first previous entry to start with the boardSpec
if (entry.Message.StartsWith(boardSpec))
{
if (entry.Message.Equals(logEntry.Message))
{
return false;
}
else
{
return true;
}
}
}
}
}
return false;
}


Any Ideas?

Regards
Andreas

Edited 2 Months Ago by LogViewPlus Support
LogViewPlus Support
LogViewPlus Support
Gaining Respect (162 reputation)Gaining Respect (162 reputation)Gaining Respect (162 reputation)Gaining Respect (162 reputation)Gaining Respect (162 reputation)Gaining Respect (162 reputation)Gaining Respect (162 reputation)Gaining Respect (162 reputation)Gaining Respect (162 reputation)
Group: Moderators
Posts: 129, Visits: 505
Hi Andreas,

This is a difficult one for me to debug without the underlying data.  I have looked at the code and can't see any reason why the 'additional' 1.7M log entries would have any impact.  That is the aspect of the problem I would like to focus on as your performance issue only happens with the 'additional' records.

If you are filtering off of a parent filter with 8000 log entries, then I would expect the Show method to be called 8000 times.  I would also expect logEntry.FindPrevious(LookupSource.ParentFilter) never returns more then 8000 entries.  Can you please confirm this assumption is correct by adding logging or trace statements?

I would also expect the first call into logEntry.FindPrevious takes time, but subsequent requests (for the same source LogEntry) should be very quick.  It may be worth adding a timer to see how long that first call takes.  This is the timer I frequently use:

        private static System.Diagnostics.Stopwatch _stopWatch = null;
private static void StartTimer()
{
_stopWatch = System.Diagnostics.Stopwatch.StartNew();
}

private static string StopTimer()
{
_stopWatch.Stop();
return _stopWatch.Elapsed.ToString();
}


Thanks,

Toby
GO

Merge Selected

Merge into selected topic...



Merge into merge target...



Merge into a specific topic ID...




Similar Topics

Login

Explore
Messages
Mentions
Search