I'm trying to work with high volume logs using `Merged Logs` (as I need to combine logs from different sources into one) that reaches 10 000 entries per minute. LogViewPlus is very often stuck in `Unresponsive` state or is very slow the respond to mouse actions.

I've got 7 GB of free RAM on a 12 GB system according to Task Manager.
The process LogViewPlus.exe is consuming ~3 GB of private working Memory and 50% CPU load on Xeon Platinum 8160

Hi Dennis,

LogViewPlus stores two copies of every log entry in memory as UTF-16 - a parsed and unparsed version.  This makes it a memory intensive application.  I would expect a 1 GB ASCII log file to take up 4 GB in memory once parsed.  This memory will only be released when a log file is closed. Rolling log files do not release the memory.

So the high memory usage is not necessarily indicative of a problem. 

The CPU usage does sound high.  To recreate this issue locally, could you provide me with more information about how you are using LogViewPlus?  Is the application just parsing, or are filter / SQL statements being applied as well?  Do you have any rules set?  If possible, it would help if you were able to isolate the issue to a functional area.

Hope that helps,

Toby

I'm trying out two alternatives on an on-prem SharePoint 2016 Farm consisting of four servers

Option 1 is to tapping into the native ULS-log files of SharePoint.

Option 2 is to use the cmdlet Get-SPLogEvent | ConvertTo-CSV

Each log file is about 50 MB large.

Are you tailing the log files?  How many total log entries do you have open?  You can see this in the statistics grid in the lower left of the application.  50 MB doesn't sound very big if you are writing 10,000 log entries per minute.

Also, what happens if you refresh the log file?  LogViewPlus frees log entry memory on file close (or refresh), but it is up to Windows to collect the freed memory.  LogViewPlus does not force garbage collection.

Yes, tailing is one of the main benefits of using LogViewPlus.

Ok, I'll count the next time. But the avarge of three hours of logs is 46 log files with a combined size of about 1,5 GB, at the moment. I've still haven't added the logfiles of the three other SharePoint servers.
But I really can't reload the logfile (is that refreshing) once LogViewPlus is frozen...

Yes - reloading is the same as refreshing.

I suspect LogViewPlus is frozen because it is resource starved.  What I would like to do is find out why / if this resource starvation is happening.  Hopefully, you can see the performance degrading over time and narrow down where the problem might be.

Well, isn't that because the log files collected into RAM is getting to large?

Yes - that seems to be the case.  LogViewPlus is resource starved because you are asking it to do a lot of work.  If you are happy with that as the explanation, then I think we can close this ticket.  However, if my understanding of the issue is incorrect, we should keep investigating.

If you see my original reply, LogViewPlus is a memory intensive program.  If you are saying a 50 MB log file takes 7 GB in memory, that sounds like a problem that I have not yet been able to recreate.  From this discussion, it seems more likely you are tailing the log files, so the number of log entries in memory far exceeds the 50 MB in the log file.

If 46 log files is 1.5 GB on the file system, this would probably be 3 GB as UTF-16.  We need to double this because LogViewPlus stores a parsed and unparsed version of the message, so 6 GB.  You are reporting 7 GB, so that seems like expected behaviour.

Also, if LogViewPlus is consuming 7 GB and you refresh the file, Windows will not necessarily free the 7 GB.  From Windows perspective, you may have plenty of memory and freeing the 7 GB will consume CPU resources.  So it may not bother until it is necessary.

The CPU usage is harder to explain, but if you are tailing 10,000 log entries per minute, between what your application is doing to create the log entries and what LogViewPlus is doing to read / process them, that might do it.

Hope that helps clarify,

Toby