How to process optional columns in DsvParser


Author
Message
soko
soko
New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)
Group: Forum Members
Posts: 3, Visits: 5
Hi,

I'm using the latest version of LogViewPlus and I'm having trouble finding the right settings for a DsvParser... or even maybe I need a different one.

Here's my log data:
12399|2025-10-16T02:26:17.4405537+00:00|INFO|37|Lifetime|Application is shutting down...
12401|2025-10-16T02:26:17.4581051+00:00|DEBUG|46|Host|Hosting stopping
12403|2025-10-16T02:26:17.4718160+00:00|DEBUG|40|Host|Hosting stopped
59|2025-10-16T02:26:41.4117132+00:00|INFO|2|VirusScannerService|VirusScannerService|TestIfItIsWorking|Starting Test...
61|2025-10-16T02:26:41.4138900+00:00|INFO|2|VirusScannerService|VirusScannerService|.ctor|Starting to initialise AMSI


I use this CsvParser: %s|%d{yyyy-MM-ddT%H:mm:ss.fffffffzzzz}|%p|%t|%c|%m%n

as a workaround but the last two lines only show "VirusScannerService" as message and not the full text "VirusScannerService|TestIfItIsWorking|Starting Test..."

What I really want is to somehow tell the parser there are optional columns and the last column is the message.
So for the first three lines the parser of above works fine.
But for the last two it should be: %s|%d{yyyy-MM-ddT%H:mm:ss.fffffffzzzz}|%p|%t|%c|%S{class}|%S{method}|%m%n

Is something like this possible? Or do I have to use a different parser when some columns are missing in some lines?

May I even need to use a https://www.logviewplus.com/docs/custom_parsers.html or a https://www.logviewplus.com/docs/post_processors.html

At the moment I'm using this RegexParser as it gives me the optional columns in the message:
 (?<sequence>\d+)\|(?<date>[0-9T:\.\+\-]+)\|(?<level>[A-Z]+)\|(?<thread>\d+)\|(?<logger>[^|]+)\|(?<message>.*)$

thanks
Soko

PS: There is also the possibility the message has multiple lines like shown below. But it seems this seems to work with my current RegExParser:
449|2025-10-16T11:20:51.1474936+00:00|DEBUG|11|Query|Compiling query expression:
'DbSet<Project>()
  .Include(p => p.Users)
  .ThenInclude(pu => pu.User)
  .OrderBy(p => p.Code)'
451|2025-10-16T11:20:51.1548535+00:00|DEBUG|11|Query|Including navigation: 'Project.Users'.

Edited Last Week by soko
LogViewPlus Support
LogViewPlus Support
Supreme Being (12K reputation)Supreme Being (12K reputation)Supreme Being (12K reputation)Supreme Being (12K reputation)Supreme Being (12K reputation)Supreme Being (12K reputation)Supreme Being (12K reputation)Supreme Being (12K reputation)Supreme Being (12K reputation)
Group: Moderators
Posts: 1.3K, Visits: 4.4K
Hi Soko,

That's a classic scenario when log formats vary slightly. You are correct that a single DsvParser pattern will struggle because it expects a fixed number of columns, as optional columns are not supported within a single pattern.

However, you can achieve the result you want by using multiple parser configurations. This is much simpler than writing a custom parser or post-processor.

It's important to note that the Multi-Pattern Parser is not available in the Parser Wizard and must be configured manually.  Your parser configuration will look something like:



LogViewPlus will then try to match each line against the first pattern. This will correctly parse your more detailed VirusScannerService lines. If that pattern fails to match (which it will for your first three lines), it will fall back and try the second pattern, which will parse them correctly.  Correct processing depends on failing when a pattern is not matched, so it is important to list your configurations from most to least sophisticated (number of parameters).

This gives you the "optional columns" behavior you're looking for:



Multiple lines in the message should not be a problem.  That is standard behaviour for the message (%m) column.

Hope that helps!

Toby
Edited Last Week by LogViewPlus Support
soko
soko
New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)
Group: Forum Members
Posts: 3, Visits: 5
Hi Toby,
and thanks for the quick reply. I thought there must be an "out of the box" way for this scenario Smile

But the output of your solution is not completely correct. For the first three lines the message is shown in the "class" column. But the class & method column should be empty for those and i.e. "Application is shutting down..." should be in the Message column.

Is that possible?
thanks again! Soko
Edited Last Week by soko
LogViewPlus Support
LogViewPlus Support
Supreme Being (12K reputation)Supreme Being (12K reputation)Supreme Being (12K reputation)Supreme Being (12K reputation)Supreme Being (12K reputation)Supreme Being (12K reputation)Supreme Being (12K reputation)Supreme Being (12K reputation)Supreme Being (12K reputation)
Group: Moderators
Posts: 1.3K, Visits: 4.4K
Hi Soko,

Thanks for highlighting this.  That appears to be an issue with the DSV parser which I will need to investigate.

In the meantime, try using the PatternParser instead.  Your configuration can remain the same - just switch the parser:





Thanks again,

Toby

soko
soko
New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)New Member (6 reputation)
Group: Forum Members
Posts: 3, Visits: 5
Hi again. Perfect BigGrin Now it works like expected. Thanks for the help.
LogViewPlus Support
LogViewPlus Support
Supreme Being (12K reputation)Supreme Being (12K reputation)Supreme Being (12K reputation)Supreme Being (12K reputation)Supreme Being (12K reputation)Supreme Being (12K reputation)Supreme Being (12K reputation)Supreme Being (12K reputation)Supreme Being (12K reputation)
Group: Moderators
Posts: 1.3K, Visits: 4.4K
Glad that helped - thanks for letting me know!

I have made a note to investigate the DsvParser issue before the next release.  Thanks for spotting that one.

Toby
GO

Merge Selected

Merge into selected topic...



Merge into merge target...



Merge into a specific topic ID...




Similar Topics

Login

Explore
Messages
Mentions
Search