In our instrument logs, at various points we get a message like this:

DATACONNECTION - HWBytes:15,301,020 BPS:0 EvtFmBytes:15,001 HWEvents:15,001 EQE:True EQC:0 ListenRun:True ProcessRun:True DS:1020 SA:0 FES:True Name:FinalEventSeen

As shown in your video I would like to use SQL to pull out or even line graph the "EvtFmBytes:" and "HWEvents:" from these lines.

The automatic parser didn't do anything.

I tried the pattern parser but I am not sure it is dealing with the commas in the numbers correctly.   It seems to think it is a string and instead of finding similar columns it is just matching the one exact string.

We also have these lines:
SystemHealthTriggerSnapshot - Trig:AverageCpuLoad AverageCpuLoad - CPU:54% Speed:0GHz RAM:T-19.23GB/P-1.36GB/O-17.87GB Client:1%/424.93MB Server:48%/971.44MB Disk:B-235.53GB/P-235.53GB Net:A02919AAC2B0:R-8.03KB/S-1.51KB,005056C00001:R-0B/S-0B,005056C00008:R-0B/S-0B,8038FBA68B5A:R-0B/S-0B,8038FBA68B5E:R-0B/S-0B LoopAvg:206 LoopMax:2235 LoopMin:10

In a similar fashion I'd like to pull out CPU,  Data out of the "Server:" area which is CPU%/RAM used.

Some help to get started would be very much appreciated.

LogViewPlus views log entries as time series data.  In this case, your log entries do not appear to have a timestamp and therefore cannot be parsed by LogViewPlus.

If you are able to add a timestamp to the log entries, LogViewPlus should be able to parse them.  In this case, the trick is to parse the log entries separately from the log message.  It looks like most of the sample log entries provided would be treated as a log message - %m.  In other words, there does not appear to be much shared metadata between the two samples.

Hope that helps,

Toby

Toby, I omitted the timestamp.  That is just the message part.  The full entries look like this

2023-02-13 08:07:59.2512|2251|Trace|BigfootServer|NoUser|LogStatus-378|BIGFOOTDATACONNECTION - HWBytes:15,301,020 BPS:0 EvtFmBytes:15,001 HWEvents:15,001 EQE:True EQC:0 ListenRun:True ProcessRun:True DS:1020 SA:0 FES:True Name:StopAcquisitionProcess - Before unload sample

Here is the other one:

2023-02-13 15:20:29.4822|3878|Trace|BigfootServer|NoUser|LogSnapshot-666|SystemHealthTriggerSnapshot - Trig:AverageCpuLoad AverageCpuLoad - CPU:61% Speed:0GHz RAM:T-19.13GB/P-948.88MB/O-18.2GB Client:0%/333.06MB Server:47%/615.83MB Disk:B-235.52GB/P-235.52GB Net:A02919AAC2B0:R-52.76KB/S-4.84KB,005056C00001:R-0B/S-0B,005056C00008:R-0B/S-0B,8038FBA68B5A:R-0B/S-0B,8038FBA68B5E:R-0B/S-0B LoopAvg:212 LoopMax:2235 LoopMin:10

In the first case I would love to be able to graph as you do in your short video example the "EvtFmBytes:" data and the "HWEvents" data

In the second one being able to pull out CPU would be nice.  But also there are two other sections of interest:

Server:47%/615.83MB
and
Client:0%/333.06MB

In the ideal world it would be nice to pull from both of these the CPU% which is the first number before the / and the second number after the slash which is the RAM used.

mark

Thanks for the sample log entries. 

If I put these log entries together, LogViewPlus opens the file with a default parse of: 

%d{yyyy-MM-dd %H:mm:ss.ffff}|%S{Column 2}|%p|%c|%S{Column 5}|%t|%m

This is using the DsvParser:


That looks good to me and it leaves us with messages similar to the ones you defined in your original post.  The next step is to parse the messages individually.

The first message can be parsed with:

BIGFOOTDATACONNECTION - %S EvtFmBytes:%s{EvtFmBytes} HWEvents:%s{HWEvents} %S

Breaking this down...
BIGFOOTDATACONNECTION - quickly identifies which messages I am looking for (this helps improve performance).
%S - reads and ignores data until it finds...
EvtFmBytes:%s{EvtFmBytes} - our first column.
HWEvents:%s{HWEvents} - our second column.
%S - finally, we read until the end of the message.

TIP: When defining the %S data to ignore, I just highlighted the data to ignore and selected 'Mark as Custom' without supplying a column name.  The named columns were also defined using the 'Mark as Custom' command.

The second message uses the format:

SystemHealthTriggerSnapshot - %S CPU:%s{CPU} %S Client:%S{ClientCPU}/%s{ClientMem}MB Server:%S{ServerCPU}/%s{ServerMem}MB %S

The parsing technique here is very similar to the first message, so hopefully the configuration makes sense.

So, that's how to parse the messages, the next step is to query them with SQL.  Here, I ran into a problem with the BIGFOOTDATACONNECTION message.  For some reason, LogViewPlus doesn't like the data values.  I think it is struggling to convert the number format.  This is a bug which we will be sure to address in the next release.  Apologies for this - SQL parsing is new.

The second message can be queried successfully with:

SELECT CPU, ClientCPU, ClientMem, ServerCPU, ServerMem FROM CurrentView ORDER BY Timestamp DESC

When querying, note that you must select a view which provides the necessary columns.  I have duplicated your data in my test log file, so the end result is something like:


Hope that helps,

Toby

thank you so much Toby,

That give me a place to start.

Glad that helped - thanks for letting me know.

Just a quick update to let you know that we have now released LogViewPlus 3.0.11 as a BETA release.  This release resolves the issue with the 'BIGFOOTDATACONNECTION message' discussed above.  

Hope that helps,

Toby