Is it possible to ignore invalid lines in json log

petr — Fri, 12 Mar 2021 08:46:02 GMT

I've been struggling with this issue for some time.
Some of the lines in our log file (SeriLog json) are invalid (cutoff by syslog)
Every time I load up the file to LogView it crashes (standard error reporter window).

Is there a way to make logivew skip the lines? Even better would be to display them as text but that's not necessary.

Thanks

RE: Is it possible to ignore invalid lines in json log

LogViewPlus Support — Fri, 12 Mar 2021 08:46:02 GMT

Glad that helped. Thanks for the crash report - I will take a look.

Yes - the rsyslog configuration is the real pain point. It's good you were able to make this change, but I appreciate that this is not always possible for all users. I will keep thinking about this problem and see what I can come up with.

Thanks for highlighting this issue!

Toby

RE: Is it possible to ignore invalid lines in json log

petr — Fri, 12 Mar 2021 08:37:10 GMT

Thank you for the explanation and advice. I didn't realize there were logs with multiline json. You suggestion makes sense to me, it can be generalized to multi-line vs single-line parsing and it could be useful. Although I've already solved my issue by finally reconfiguring rsyslog on prod (rsyslog config is the real pain point here not logviewplus :) ).

I'd just sent the crash report (2021-03-12 09:30 UTC+1)

RE: Is it possible to ignore invalid lines in json log

LogViewPlus Support — Fri, 12 Mar 2021 07:58:36 GMT

Thanks for the sample log entry.

I was not able to recreate the 'crash' on my end. If you have not done so already, can you please click the 'Yes' button on the dialog above? This will crash the application and send us an exception report. Often we can solve these kinds of issues by looking at the exception details.

It makes sense that the file is not parsing because the log entry is invalid. The invalid JSON actually confuses LogViewPlus and makes it difficult to know where the log entry ends and the next one begins. Note that other log files may have multi-line JSON and the parser needs to be able to handle this scenario too.

The only work around I can think of is to have a setting which tells the parser "I expect all JSON to be on a single log line". However, this setting would be confusing for a lot of users. I would like to be able to handle this scenario, but I am short on ideas on how to do so.

You might want to look at writing a custom parser. The fact that the JSON is all on one line would make the implementation pretty easy. A custom parser would allow you flexibility in handling exceptions.

What I would suggest is to use the PatternParser instead. I can parse this log entry with:
[code]{"Timestamp":"%d{yyyy-MM-ddT%H:mm:ss.fffffffzzzz}","Level":"%p","MessageTemplate":"%S{MessageTemplate}","%S":"%m%n[/code]

Note that the configuration above ignores the name of the final JSON tag. It simply takes the last element as the log entry message and does not look for a close.

The downside of this approach is that the log entries will not be pretty printed as JSON, you will need to change the pretty print style manually:

Hope that makes sense.

Toby

RE: Is it possible to ignore invalid lines in json log

petr — Thu, 11 Mar 2021 18:14:26 GMT

Thanks for replying.
I was wrong, it's not standard crash dialog.

This is the dialog.

This is the line that crashes it. This line is too long for syslog and it cuts off the json midway.

[code]{"Timestamp":"2021-03-09T14:58:44.5744442+01:00","Level":"Error","MessageTemplate":"An unhandled exception has occurred while executing the request.","Exception":"NHibernate.Exceptions.GenericADOException: could not execute query\n[ select campaign0_.campaign_id as campaign1_24_0_, identities1_.campaign_id as campaign1_29_1_, identities1_.guest_identity_id as guest2_29_1_, campaign0_.company_department_id as company2_24_0_, campaign0_.name as name3_24_0_, campaign0_.channel as channel4_24_0_, campaign0_.mode as mode5_24_0_, campaign0_.pickup_rate as pickup6_24_0_, campaign0_.created as created7_24_0_, campaign0_.created_by_user_id as created8_24_0_, campaign0_.started as started9_24_0_, campaign0_.finished as finished10_24_0_, campaign0_.routing_rule_set_id as routing11_24_0_, campaign0_.contact_info_field as contact12_24_0_, campaign0_.call_script_form_id as call13_24_0_, campaign0_.trigger_set_id as trigger14_24_0_, campaign0_.bcw_limit as bcw15_24_0_, identities1_.state as state3_29_1_, identities1_.email_thread_id as email4_29_1_, identities1_.retry_scheduled_date as retry5_29_1_, (select ca.campaign_call_attempt_id from campaign_call_attempt as ca where ca.guest_identity_id = identities1_.guest_identity_id and ca.campaign_id = identities1_.campaign_id order by ca.created desc limit 1) as formula1_1_, identities1_.campaign_id as campaign1_29_0__, identities1_.guest_identity_id as guest2_29_0__ from campaign campaign0_ left outer join campaign_identity identities1_ on campaign0_.campaign_id=identities1_.campaign_id where campaign0_.company_department_id=:p0 and campaign0_.campaign_id=:p1 ]\n Name:p1 - Value:2002000 Name:p2 - Value:190\n[SQL: select campaign0_.campaign_id as campaign1_24_0_, identities1_.campaign_id as campaign1_29_1_, identities1_.guest_identity_id as guest2_29_1_, campaign0_.company_department_id as company2_24_0_, campaign0_.name as name3_24_0_, campaign0_.channel as channel4_24_0_, campaign0_.mode as mode5_24_0_, campaign0_.pickup_rate as pickup6_24_0_, campaign0_.created as created7_24_0_, campaign0_.created_by_user_id as created8_24_0_, campaign0_.started as started9_24_0_, campaign0_.finished as finished10_24_0_, campaign0_.routing_rule_set_id as routing11_24_0_, campaign0_.contact_info_field as contact12_24_0_, campaign0_.call_script_form_id as call13_24_0_, campaign0_.trigger_set_id as trigger14_24_0_, campaign0_.bcw_limit as bcw15_24_0_, identities1_.state as state3_29_1_, identities1_.email_thread_id as email4_29_1_, identities1_.retry_scheduled_date as retry5_29_1_, (select ca.campaign_call_attempt_id from campaign_call_attempt as ca where ca.guest_identity_id = identities1_.guest_identity_id and ca.campaign_id = identities1_.campaign_id order by ca.created desc limit 1) as formula1_1_, identities1_.campaign_id as campaign1_29_0__, identities1_.guest_identity_id as guest2_29_0__ from campaign campaign0_ left outer join campaign_identity identities1_ on campaign0_.campaign_id=identities1_.campaign_id where campaign0_.company_department_id=:p0 and campaign0_.campaign_id=:p1]\n ---> Npgsql.PostgresException (0x80004005): 42703: column identities1_.email_thread_id does not exist\n at Npgsql.NpgsqlConnector.<>c__DisplayClass161_0.<<ReadMessage>g__ReadMessageLong|0>d.MoveNext()\n--- End of stack trace from previous location where exception was thrown ---\n at Npgsql.NpgsqlConnector.<>c__DisplayClass161_0.<<ReadMessage>g__ReadMessageLong|0>d.MoveNext()\n--- End of stack trace from previous location where exception was thrown ---\n at Npgsql.NpgsqlDataReader.NextResult(Boolean async, Boolean isConsuming)\n at Npgsql.NpgsqlDataReader.NextResult()\n at Npgsql.NpgsqlCommand.ExecuteDbDataReader(CommandBehavior behavior, Boolean async, CancellationToken cancellationToken)\n at Npgsql.NpgsqlCommand.ExecuteDbDataReader(CommandBehavior behavior)\n at NHibernate.AdoNet.AbstractBatcher.ExecuteReader(DbCommand cmd)\n at NHibernate.Loader.Loader.GetResultSet(DbCommand st, QueryParameters queryParameters, ISessionImplementor session, IResultTransformer forcedResultTransformer)\n at NHibernate.Loader.Loader.DoQuery(ISessionImplementor session, QueryParameters queryParameters, Boolean returnProxies, IResultTransformer forcedResultTransformer)\n at NHibernate.Loader.Loader.DoQueryAndInitializeNonLazyCollections(ISessionImplementor session, QueryParameters queryParameters, Boolean returnProxies, IResultTransformer forcedResultTransformer)\n at NHibernate.Loader.Loader.DoList(ISessionImplementor session, QueryParameters queryParameters, IResultTransformer forcedResultTransformer)\n --- End of inner exception stack trace ---\n at NHibernate.Loader.Loader.DoList(ISessionImplementor session, QueryParameters queryParameters, IResultTransformer forcedResultTransformer)\n at NHibernate.Loader.Loader.ListIgnoreQueryCache(ISessionImplementor session, QueryParameters queryParameters)\n at NHibernate.Hql.Ast.ANTLR.QueryTranslatorImpl.List(ISessionImplementor session, QueryParameters queryParameters)\n at NHibernate.Engine.Query.HQLQueryPlan.PerformList(QueryParameters queryParameters, ISessionImplementor session, IList results)\n at NHibernate.Impl.SessionImpl.List(IQueryExpression queryExpression, QueryParameters queryParameters, IList results, Object filterConnection)\n at NHibernate.Impl.AbstractSessionImpl.List(IQueryExpression queryExpression, QueryParameters parameters)\n at NHibernate.Impl.AbstractQueryImpl2.List()\n at NHibernate.Linq.DefaultQueryProvider.ExecuteQuery(NhLinqExpression nhLinqExpression, IQuery query, NhLinqExpression nhQuery)\n at NHibernate.Linq.DefaultQueryProvider.Execute(Expression expression)\n at NHibernate.Linq.DefaultQueryProvider.Execute[TResult](Expression expression)\n at iCord.OnlineOperator.Api.Admin.Campaign.CampaignsController.CreateCampaignIdentities(TenantContext context, Int32 id, CampaignUpdateIdentitiesFromCrmModel model, CancellationToken cancellationToken) in /opt/buildagent/work/a267bca009a26fc8/OnOpChat/iCord.OnlineOperator.Api_Core/Admin/Campaign/CampaignsController.cs:line 301\n at Microsoft.Extensions.Internal.ObjectMethodExecutor.<>c__DisplayClass33_0.<WrapVoidMethod>b__0(Object target, Object[] parameters)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.VoidResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeActionMethodAsync()\n at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeNextActionFilterAsync()\n--- End of stack trace from previous location where exception was thrown ---\n at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeInnerFilterAsync()\n--- End of stack trace from previous location where exception was thrown ---\n at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResourceFilter>g__Awaited|24_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeFilterPipelineAsync()\n--- End of stack trace from previous location where exception was thrown ---\n at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Logged|17_1(ResourceInvoker invoker)\n at Microsoft.AspNetCore.Builder.RouterMiddl[/code]

RE: Is it possible to ignore invalid lines in json log

LogViewPlus Support — Thu, 11 Mar 2021 17:40:05 GMT

Hi Petr,

That's interesting - I have not seen this problem before.

Currently, it is not possible without resorting to your own customer parser. LogViewPlus expects JSON log entries to be formatted as JSON. You might have better luck if you use the PatternParser instead as in this case, your log entries are not all JSON,

Are you able to send me a sample log file? If you contact us, I can send you my email address. I might be able to help if I could see the detail. I am also interested in why LogViewPlus is crashing. Not parsing the file is one thing - but an application crash should not happen.

Hope that helps,

Toby

LogViewPlus Support » LogViewPlus Support » Help & Support » Is it possible to ignore invalid lines in json log

Is it possible to ignore invalid lines in json log

RE: Is it possible to ignore invalid lines in json log

RE: Is it possible to ignore invalid lines in json log

RE: Is it possible to ignore invalid lines in json log

RE: Is it possible to ignore invalid lines in json log

RE: Is it possible to ignore invalid lines in json log