Pattern parser takes multiple entries as one entry

asaf — Tue, 24 Sep 2019 13:53:01 GMT

Hello,
I just discoveredLogViewPlus and it seems very promising!
I installed an evaluation copy and I'm trying to use it to parse our logs.

I've encountered three issues so far:
1) It seems the automatic pattern parser takes more than 1 log entry into an entry.
Meaning that sometimes entries are incorrectly clumped together.
For example the following 2 entries are detected as 1 entry:
01/06/2019 12:18:52.160 [ERROR] [scr.indexes.CalculateIndexesQueue] (EJB default - 6) [634d4406-0fbd-4936-a332-78314566bfcc] Failed to add algoData (ID: 131) to indexCalculationReadinessQueue: javax.jms.JMSRuntimeException: Could not create a session: IJ000460: Error checking for a transaction
    at org.apache.activemq.artemis.ra.ActiveMQRAConnectionFactoryImpl.createContext(ActiveMQRAConnectionFactoryImpl.java:428)
    at org.apache.activemq.artemis.ra.ActiveMQRAConnectionFactoryImpl.createContext(ActiveMQRAConnectionFactoryImpl.java:410)
    at org.apache.activemq.artemis.ra.ActiveMQRAConnectionFactoryImpl.createContext(ActiveMQRAConnectionFactoryImpl.java:405)
    at scr.utils.Jms.send(Jms.java:80)
    at scr.utils.Jms.Send(Jms.java:29)
    at scr.indexes.CalculateIndexesQueue.EnqueueIfAllowed(CalculateIndexesQueue.java:84)
    at scr.indexes.CalculateIndexesQueue.EnqueueAllIndexes(CalculateIndexesQueue.java:58)
    at scr.init.Scheduler.scheduledCalculateIndexes(Scheduler.java:344)
    at sun.reflect.GeneratedMethodAccessor490.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52)
    at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
    at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:57)
    at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:61)
    at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
    at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:437)
    at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.delegateInterception(Jsr299BindingsInterceptor.java:70)
    at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:80)
    at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:95)
    at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:61)
    at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
    at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:57)
    at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:61)
    at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
    at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:437)
    at org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:73)
    at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:83)
    at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
    at org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45)
    at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
    at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21)
    at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
    at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
    at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:52)
    at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
    at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53)
    at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
    at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:275)
    at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:327)
    at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:239)
    at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
    at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
    at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
    at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100)
    at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
    at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64)
    at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
    at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
    at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
    at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:54)
    at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
    at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:64)
    at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
    at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
    at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:636)
    at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:61)
    at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
    at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
    at org.jboss.invocation.PrivilegedWithCombinerInterceptor.processInvocation(PrivilegedWithCombinerInterceptor.java:80)
    at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
    at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
    at org.jboss.as.ejb3.timerservice.TimedObjectInvokerImpl.callTimeout(TimedObjectInvokerImpl.java:104)
    at org.jboss.as.ejb3.timerservice.CalendarTimerTask.invokeBeanMethod(CalendarTimerTask.java:64)
    at org.jboss.as.ejb3.timerservice.CalendarTimerTask.callTimeout(CalendarTimerTask.java:53)
    at org.jboss.as.ejb3.timerservice.TimerTask.retryTimeout(TimerTask.java:208)
    at org.jboss.as.ejb3.timerservice.TimerTask.run(TimerTask.java:163)
    at org.jboss.as.ejb3.timerservice.TimerServiceImpl$Task$1.run(TimerServiceImpl.java:1221)
    at org.wildfly.extension.requestcontroller.RequestController$QueuedTask$1.run(RequestController.java:497)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
    at org.jboss.threads.JBossThread.run(JBossThread.java:320)
Caused by: javax.jms.JMSException: Could not create a session: IJ000460: Error checking for a transaction
    at org.apache.activemq.artemis.ra.ActiveMQRASessionFactoryImpl.allocateConnection(ActiveMQRASessionFactoryImpl.java:895)
    at org.apache.activemq.artemis.ra.ActiveMQRASessionFactoryImpl.createSession(ActiveMQRASessionFactoryImpl.java:520)
    at org.apache.activemq.artemis.ra.ActiveMQRASessionFactoryImpl.createSession(ActiveMQRASessionFactoryImpl.java:734)
    at org.apache.activemq.artemis.ra.ActiveMQRASessionFactoryImpl.createSession(ActiveMQRASessionFactoryImpl.java:739)
    at org.apache.activemq.artemis.ra.ActiveMQRAConnectionFactoryImpl.validateUser(ActiveMQRAConnectionFactoryImpl.java:467)
    at org.apache.activemq.artemis.ra.ActiveMQRAConnectionFactoryImpl.createContext(ActiveMQRAConnectionFactoryImpl.java:420)
    ... 72 more
Caused by: javax.resource.ResourceException: IJ000460: Error checking for a transaction
    at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:424)
    at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:747)
    at org.apache.activemq.artemis.ra.ActiveMQRASessionFactoryImpl.allocateConnection(ActiveMQRASessionFactoryImpl.java:853)
    ... 77 more
Caused by: javax.resource.ResourceException: IJ000459: Transaction is not active: tx=TransactionImple < ac, BasicAction: 0:ffff7f000101:-8ada536:5ceec76e:81d976 status: ActionStatus.ABORTED >
    at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:408)
    ... 79 more

01/06/2019 12:18:52.363 [INFO ] [scr.reports.engine.ReportCacheManager] (EJB default - 8) [4f135661-a60e-4c4b-b1b4-147a5254c295] Refresh Calculation Cache by Timer: PantingTrendCalculation

I couldn't find a way to solve this.

2) Also using the same example as above, I'd love if the exception will appear in an Exception column. and not in the Message column.

3) Grouping entries is a very powerful feature. However many times messages that should be grouped together still have very small differences. For example messages saying "Attempt number X failed". I'd like those messages grouped even if the attempt number is different.
Is it possible to achieve this?
Perhaps messages can be grouped when the number of differences between them is below some threshold?

Many thanks,
Asaf

RE: Pattern parser takes multiple entries as one entry

LogViewPlus Support — Tue, 24 Sep 2019 13:53:01 GMT

Thanks for the feedback.

We actually have a few ideas on how we can improve here, but unfortunately it will probably be early next year before we get a chance to action them. In the meantime I will take a look at improving the in-application documentation to highlight this functionality.

Thanks,

Toby

RE: Pattern parser takes multiple entries as one entry

IVTH — Tue, 24 Sep 2019 13:08:53 GMT

I was thinking about posting on mixed parsing styles in one file, and then I found this! This makes the produce much more usable on many of my logs! Maybe I just missed it glancing through the documentation, but having known about the multi-pattern parsing earlier would have been great.

RE: Pattern parser takes multiple entries as one entry

LogViewPlus Support — Wed, 05 Jun 2019 13:15:37 GMT

That's great Asaf - thanks for letting me know!

Toby

RE: Pattern parser takes multiple entries as one entry

asaf — Wed, 05 Jun 2019 13:12:07 GMT

Wonderful, multi pattern seem to have solved the issue!

RE: Pattern parser takes multiple entries as one entry

LogViewPlus Support — Wed, 05 Jun 2019 13:01:11 GMT

Yes - that could definitely be the problem. As I mentioned, LogViewPlus does not currently support optional fields.

What you could try in this case is a multi-pattern as described here:
https://www.logviewplus.com/docs/pattern_parser.html

Please try the pattern:
%d{dd/MM/yyyy %H:mm:ss.fff} [%p] [%c] (%t) [%s{CorrelationId}] %m%n
%d{dd/MM/yyyy %H:mm:ss.fff} [%p] [%c] (%t) %m%n

Note the lower case 's' on CorrelationId.

Multi-patterns work best when the date field is in a different format, but it is definitely worth trying.

The RegexParser is also worth trying, but unfortunately my regex knowledge is a work in progress. :-) I may not be much help in building the correct regex.

Thanks,

Toby

RE: Pattern parser takes multiple entries as one entry

asaf — Wed, 05 Jun 2019 12:40:43 GMT

Thanks, I'll try.
I think part of the problem is that some of the log entries don't have CorrelationId, so they don't fit the patter.
Is there a way to make LogViewPlus handle such entries?
Perhaps the Regex Parser will do better? (Although I have no idea how to set it up)

RE: Pattern parser takes multiple entries as one entry

LogViewPlus Support — Wed, 05 Jun 2019 12:34:41 GMT

Thanks Asaf,

That pattern looks OK me. Very strange. I will need to take a closer look and get back to you later today.

In the meantime, as a temporary solution, I would recommend you simplify the pattern until the you get a successful parse. For example,%d{dd/MM/yyyy %H:mm:ss.fff} [%p] [%c] (%t) %m%n

Thanks,

Toby

RE: Pattern parser takes multiple entries as one entry

asaf — Wed, 05 Jun 2019 11:26:58 GMT

Hi Toby,
Thanks for the quick response.

This is the Pattern I'm using to parse the file:
%d{dd/MM/yyyy %H:mm:ss.fff} [%p] [%c] (%t) [%S{CorrelationId}] %m%n

Thanks,
Asaf

RE: Pattern parser takes multiple entries as one entry

LogViewPlus Support — Wed, 05 Jun 2019 10:12:52 GMT

Hi Asaf,

Thanks for taking the time to evaluate LogViewPlus.

The first issue is fairly common and indicates a problem with your parser configuration. Usually, this occurs when the date format is wrong, or the log entry is not in the format expected. For example, it may be because the exception entry does not have the expected number of columns.

Can you please send me more information on how you are parsing this file? In particular, the configuration pattern you are using.

The second issue is tricky. Not all of your log entries have an exception and there does not appear to be any clear way to differentiate between messages and exceptions. Also, LogViewPlus does not support optional fields. So I don't think this will be possible for this log file.

Currently, LogViewPlus cannot take variations into account when grouping. What I would recommend is using a filter to isolate these entries and then creating sub-filters as needed. I will have a look into how we can improve this in the future.

Hope that helps. Please let me know if you have any further questions or issues.

Thanks,

Toby

LogViewPlus Support » LogViewPlus Support » Help & Support » Pattern parser takes multiple entries as one entry

Pattern parser takes multiple entries as one entry

RE: Pattern parser takes multiple entries as one entry

RE: Pattern parser takes multiple entries as one entry

RE: Pattern parser takes multiple entries as one entry

RE: Pattern parser takes multiple entries as one entry

RE: Pattern parser takes multiple entries as one entry

RE: Pattern parser takes multiple entries as one entry

RE: Pattern parser takes multiple entries as one entry

RE: Pattern parser takes multiple entries as one entry

RE: Pattern parser takes multiple entries as one entry