LogViewPlus Support » LogViewPlus Support » Help & Support » JSON Parsing Error

JSON Parsing Error

SoBeGuy — Sat, 30 Apr 2022 16:23:59 GMT

Just sent you an error report generated by LVP when it failed to parse my JSON file.

I'm guessing the parsing error is happening beause one of the JSON properties contains encoded JSON, which is somehow confusing the parser. The encoded JSON is properly escaped, so it really should be handled the same as any other string value.

By the way, the reason I'm encoding JSON in a JSON object property is beause I need to log an API response that's sent as JSON. Unfortuntaely, your JSON parser only supports primitives, which kind of defeats the purpose of using JSON. Ideally, I could take the API response and store it as a proper JSON object within a property, instead of having to encode it to a string.

RE: JSON Parsing Error

LogViewPlus Support — Sat, 30 Apr 2022 16:23:59 GMT

Hi AV,

I investigated this issue today and it is going to be more difficult to resolve than I initially expected. We will not be able to produce a solution in the short term.

There are two main problems that we have at the moment.
1. The Parse Message command wants to identify other, similar messages in the log file. Dealing with a variable like 'JSON' is too ambiguous. For example, if multiple messages have a lot of differentiation, what would the column headings be?
2. There is a performance impact to parsing JSON (mainly from a memory perspective). LogViewPlus processes messages when the file is parsed. Right now, we don't to take the performance hit of parsing JSON unless we know the user was going to need this feature.

This is absolutely something that we will be investigating and adding in the future - it is just not something that we can quickly fix.

For the moment, the best solution I can suggest is to:
1. Manually identify the messages in the log file which are similar.
2. Export those log entries to a new log file.
3. Open the newly created log file. Create a new parser configuration as needed.

You mentioned "very different objects each time". Because of this, I don't think you will be able to easily solve your problem with a custom parser. All LogViewPlus parsers require a conformity with the logging format. If you have a lot of variability in your messages, this is going to be difficult. Again - what would your column headers be? Also, if your messages were more uniform, you wouldn't need a customer parser - the existing JSON parser would probably work fine.

Hope that helps,

Toby

RE: JSON Parsing Error

av2408 — Fri, 29 Apr 2022 20:03:18 GMT

I've got answer to all my questions.
Waiting for BETA.

RE: JSON Parsing Error

av2408 — Fri, 29 Apr 2022 20:00:48 GMT

Thanks a lot.
You are posting answers faster than I am completing questions ) This is amazing responsiveness, thank you.
Just for case, this is the link on the video: Parsing json in message

The question is still actual: can I achieve my result by creating custom parser?

Just for case, if this my be helpful for your team, here is my case;
i am generating logs from custom script for .Net application (NinjaTrader) with the help of Newtonsoft json converter and saving them to file using NLog.

And I need to send very different objects each time.

Thanks again. I am looking forward for BETA.

RE: JSON Parsing Error

LogViewPlus Support — Fri, 29 Apr 2022 19:47:36 GMT

Please see my comment above. I will take a look at this for the next BETA release.

This is the only way to parse these fields without creating a custom parser. I can help you with specific questions about LogViewPlus, including creating custom parsers, but I cannot help you create a custom parser.

RE: JSON Parsing Error

av2408 — Fri, 29 Apr 2022 19:44:34 GMT

"Parse message" do not extract columns (
Here is the video:[YouTube]http://www.youtube.com/watch?v=M_gcOC3pZr4[/YouTube]

Can I extract columns from json without creating custom parser in C#?
Can I extract columns from json by creating custom parser in C#? Will you help me? )

RE: JSON Parsing Error

LogViewPlus Support — Fri, 29 Apr 2022 19:40:35 GMT

We are actively working on making this happen. The Parse Message command outlined above represents our current best efforts on this approach. The next BETA release due out early next week will further enhance our message parsing capabilities.

Unfortunately, I had a look at our Parse Message unit tests in JSON scenarios and I don't think this is going to work for you at the moment. We currently treat the JSON as one block - which would not be an improvement for you. Let me take a look and see if we can improve things for the next BETA release.

RE: JSON Parsing Error

av2408 — Fri, 29 Apr 2022 19:35:57 GMT

Clarification about the question above: may LogViewPlus automatically detect key-value-pairs ? I mean variable number of key-value-pairs of variable types, without a pre-defined set of exact keys/columns?

RE: JSON Parsing Error

LogViewPlus Support — Fri, 29 Apr 2022 19:22:29 GMT

Hi Andrii,

Try right clicking on the message and selecting Parse Message:

This doesn't explicitly parse the JSON, but it should get you most of the way there. Alternatively, you can try modifying your parser configuration to provide more detail.

Hope that helps,

Toby

RE: JSON Parsing Error

av2408 — Fri, 29 Apr 2022 19:17:23 GMT

Hello.
I tried the method Brad provided.
It works fine on my side.
But, entire JSON object still parsed as a text message and placed in a one column.
May I let LogViewPlus to create columns based on json key-value-pairs like on screenshot attached?

Thank you.
Andrii

RE: JSON Parsing Error

SoBeGuy — Tue, 07 Sep 2021 19:29:27 GMT

I think the issue is that the conversion specifier is expecting a string, so it automatically escapes any quotes within the string. Currently it has no way of knowing that the message field contains JSON.

RE: JSON Parsing Error

LogViewPlus Support — Tue, 07 Sep 2021 18:10:17 GMT

Thanks Brad. I still have this thread on my todo list, but I will take a look at copying the JSON data as well.

I think the "mj" is implied if you are using the JSON parser. I suspect there is a simple way to display the string without escaping the JSON.

RE: JSON Parsing Error

SoBeGuy — Tue, 07 Sep 2021 16:22:12 GMT

Another issue is that it treats the embedded object as a string, so when you copy the contents of the Message column, all the quotes are escaped. I can work around it by copying from the raw log entry in the bottom pane, but that's a pain, because you have to manually figure out the number of indenting levels. Could you add a setting in the conversion specifer syntax, to indicate that a field is a JSON object, not a string? Maybe appending a "j" to the specifier placeholder. For example, instead of %m, you could use %mj.

RE: JSON Parsing Error

LogViewPlus Support — Tue, 03 Aug 2021 13:53:26 GMT

Thanks for bringing this issue to my attention. I can see why this was causing confusion and will take a look for the next release.

RE: JSON Parsing Error

SoBeGuy — Tue, 03 Aug 2021 13:43:49 GMT

Yes, I tested it with a property containing an ojbect and it works perfectly. I was confused about this because I couldn't figure out how to get the wizard to accept an object. When I highlighted the object, it said it was not a valid primitive. It would be nice if you could make it so the wizard would accept an object, or at least mention in the error message that you can force it to accept it by making it a string.

RE: JSON Parsing Error

LogViewPlus Support — Tue, 03 Aug 2021 08:06:20 GMT

In general, I agree that the application should not crash like that. However, even more important is that the log file is displayed correctly and no log entries are 'hidden' or 'lost'. In this case, invalid JSON was detected. LogViewPlus tried to recover by reading the next log line and combining it with the (so far) invalid JSON. When this failed, it became impossible for the application to display the file accurately. In this case, I believe a crash is appropriate.

In the next version of LogViewPlus, we plan to add an option to the JSON parser which allows for 'single line' JSON. If the JSON log entry should exist on a signal line, then the scanning process can be skipped. The JSON is still invalid, but the parser we are introducing in this release should be significantly better at handling invalid JSON. Even if the line cannot be parsed, it can still be combined with another line so that processing can continue. I believe your JSON object is on a single line, so this kind of logic would help.

To store a JSON object in a property, define your conversion specifier in the usual way:
[code language="css"]{
    "message":"%m"
}[/code]

This definition can process messages like:
[code language="css"]{
"message": "Application initializing."
}[/code]

Or:
[code language="css"]{
    "message": {
        "value1" : "Hello",
        "value2" : "World",
    }
}[/code]

It is necessary for the message field to be a string in the parser configuration (where the conversion specifier is defined). However, the field value can be any JSON type.

The above JSON will be displayed in LogViewPlus as:

Hope that helps,

Toby

RE: JSON Parsing Error

SoBeGuy — Mon, 02 Aug 2021 21:52:38 GMT

[quote]Also - it is only the parser definition that must work with JSON primitives. This is because the conversion specifier is is a string. There should be no issue with using a conversion specifier to store a JSON object. [/quote]

Can you elaborate on this?

How would I go about creating a JSON parser that could store a JSON object in a property, instead of having to convert it to a stirng?

RE: JSON Parsing Error

SoBeGuy — Mon, 02 Aug 2021 21:48:27 GMT

I figured out the problem. There was some non-JSON data on the previous line that shouldn't have been there. Nevertheless, it shouldn't crash like that. I guess the crash is coming from the JSON parser, but it should recover more elgantly like how the other parsers do, and display the dialog where you can edit the template.

RE: JSON Parsing Error

LogViewPlus Support — Mon, 02 Aug 2021 21:17:59 GMT

Thanks Brad. I have had a look at the bug report and there is nothing wrong with the JSON line that failed. This is unfortunate as it is going to make the issue harder to track down. The issue seems to be with the prior line.

What's happening here is that LogViewPlus needs to determine when a full JSON log entry has been read (keeping in mind that JSON may span multiple lines in the log file). This is done prior to JSON parsing. In this case, it is receiving a line of perfectly formatted JSON which cannot be parsed correctly because apparently something in the prior line has confused the pre-scan.

We have made a lot of JSON changes in this release and plan to make a lot more in the next release. However, in this case, I think this issue would have existed in prior versions as well. There seem to be something in the file, prior to this log line, which is causing the invalid parse.

I realize this is not always possible, but if you are able to contact me via email and send me the log file I can probably get this turned around very quickly. Regardless, I will take a closer look at the pre-scan code to see if I can determine where the error might be.

Also - it is only the parser definition that must work with JSON primitives. This is because the conversion specifier is is a string. There should be no issue with using a conversion specifier to store a JSON object.