LogViewPlus Support

json log

https://www.logviewplus.com/forum/Topic266.aspx

By drache42 - 25 Jul 2019

Hello,
I'm currently evaluating your product to determine if it will work for parsing our logs.
We have json logs that certain key/value pairs always exist and some that are only there certain times. The order of the key/value pairs could be different as well.
I'm noticing that sometimes the logs get parsed incorrectly and the values show up in the wrong column. Is this a issue in your product or the way I've setup my json parser?

Parser setup
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
{
"Timestamp": "%d{yyyy-MM-ddT%H:mm:ss.fffffffzzzz}",
"Level": "%p",
"RenderedMessage": "%m",
"Properties": {
"EventId": {
 "Id": "%S{Event Id}",
 "Name": "%S{Event Name}" },
"ProcessId": "%s{Process Id}",
"ThreadId": "%t",
"SourceContext":"%S{Source Context}"
}
}
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

Log that parses correctly
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
{
"Timestamp": "2019-07-19T09:48:52.2880729-07:00",
"Level": "Debug",
"MessageTemplate": "{task} processing completed in {duration}.",
"RenderedMessage": "\"AgentHeartbeatTask\" processing completed in 00:00:00.3968037.",
"Properties": {
 "task": "AgentHeartbeatTask",
 "duration": "00:00:00.3968037",
 "EventId": {
  "Id": 22,
  "Name": "Background task ended"
 },
 "SourceContext": "Tableau.PowerTools.Server.Common, Version=2019.3.0.0, Culture=neutral, PublicKeyToken=null",
 "Scope": [
  "ImportunateTaskRunner starting task: RunAgentHeartbeatAsync"
 ],
 "ProcessId": 476,
 "ThreadId": 11,
 "PTTaskDesc": "AgentHeartbeatTask",
 "PTCorrelationId": "a01f0ca3-5581-4b4c-9a5d-3f32490ff7ae"
}
}
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

Logs that parse incorrectly. The string "SourceContext": "Tableau.PowerTools.Server.Common, Version=2019.3.0.0, Culture=neutral, PublicKeyToken=null" ends up in the “Event ID” column when it should be in the “SourceContext” column.
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
{
"Timestamp": "2019-07-19T09:48:52.2940717-07:00",
"Level": "Debug",
"MessageTemplate": "{task} scheduled to run again in {duration} ms.",
"RenderedMessage": "\"AgentHeartbeatTask\" scheduled to run again in 300000 ms.",
"Properties": {
 "task": "AgentHeartbeatTask",
 "duration": 300000,
 "SourceContext": "Tableau.PowerTools.Server.Common, Version=2019.3.0.0, Culture=neutral, PublicKeyToken=null",
 "Scope": [
  "ImportunateTaskRunner starting task: RunAgentHeartbeatAsync"
 ],
 "ProcessId": 476,
 "ThreadId": 14
}
}
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

Thank you
By LogViewPlus Support - 25 Jul 2019

Hi,

Thanks for the detailed bug report.  I have been able to recreate the issue locally.

LogViewPlus stores all log entries into a common object that treats string (%s) columns as a list.  Column assignment in this list is based on index position.  The problem here is that when a string column is missing, the next string column is then written into the wrong index position.  Having columns parsed out of order can also cause this problem.  

This is the first time we have seen this issue.  I will make sure it gets fixed in the next release.

In the meantime, I would recommend one of two options:
1.  Do not parse fields that are optional or may be out of order.
2.  Use one of the 'Advanced Specifiers' as a placeholder on the field you want to parse.  Note that only string columns suffer from the index position issue described above.

For example, I can parse your logs successfully with the configuration:

{
 "Timestamp": "%d{yyyy-MM-ddT%H:mm:ss.fffffffzzzz}",
 "Level": "%p",
 "RenderedMessage": "%m",
 "Properties": {
  "EventId": {
   "Id": "%u",
   "Name": "%M"
  },
  "ProcessId": "%s{Process Id}",
  "ThreadId": "%t",
  "SourceContext": "%c"
 }
}


As there is only one string in this configuration, ordering will not be an issue.  Note that the Identity (%i) and Method Name (%M) columns are not used very often and will need to be added to your grid view manually.  This setting is then saved with the parser configuration, so you will only need to do this once.

A full list of advanced specifiers is available here:
https://www.logviewplus.com/docs/advanced_specifiers.html

Hope that helps.  Thanks again for reporting this issue.

Toby
By drache42 - 25 Jul 2019

Thank you, that helps a lot
By LogViewPlus Support - 28 Jul 2019

Hi Drache,

We have now released LogViewPlus v2.3.19 into BETA which addresses the issue discussed above.  Using this version, I can apply your original parsing configuration:
{
   "Timestamp": "%d{yyyy-MM-ddT%H:mm:ss.fffffffzzzz}",
   "Level": "%p",
   "RenderedMessage": "%m",
   "Properties": {
   "EventId": { 
       "Id": "%S{Event Id}", 
       "Name": "%S{Event Name}" },
      "ProcessId": "%s{Process Id}",
      "ThreadId": "%t",
      "SourceContext":"%S{Source Context}"
   }
}


LogViewPlus can now handle blank and out of order fields (see Event ID and Process ID columns).


The latest BETA release can be downloaded from:
https://www.logviewplus.com/download.html

Thanks again for bringing this problem to our attention,

Toby
By drache42 - 29 Jul 2019

I've installed the beta. However, now when I try to change the parser (without making any changes to the parsing json) I get the following error: "Parsing Failed: Object reference not set to an instance of an object".
By LogViewPlus Support - 29 Jul 2019

Hi,

Thanks for reporting this issue.  This bug was introduced as part of the change to allow optional columns and we missed it during our testing. 

I have just released LogViewPlus v2.3.21 which resolves this issue.

Thanks again,

Toby