LogViewPlus Support » LogViewPlus Support » Help & Support » Watch the most recent folder

Watch the most recent folder

samaursa — Tue, 26 Apr 2022 00:22:40 GMT

Hello,

I am trying to use the watch folder feature but not having luck configuring it such that it watches the most recent folder that was created. The way out logs work is that each launch of the application results in a new folder with several (not just one) log files inside.

Each log file inside has a different name. I am interested in _some_ but not all the the log files.

These are the folders

Inside one of the folders are all sorts of different logs. I only want a subset of them to be opened

Ideally what I would like is logviewplus to open a subset of the files in the latest folder (as seen above) and merge the files.

As it stands, I have to manually open the files every single time, and because there are many different files I have to be careful in dragging/dropping only the files that I know I want (I also see no option to have logviewplus skip opening files that I ask it to always ignore in a particular workspace).

Any chance I can wrangle these logs better?

RE: Watch the most recent folder

samaursa — Tue, 26 Apr 2022 00:22:40 GMT

Wildcard matching with multiple filenames will improve things a lot and I think will get me there almost all the way.

[quote]I think we should aim to include this as well by defaulting to the name of the directory that matched by the wildcard.[/quote]

Do you mean the first matched directory? If yes, then that would work however it depends on how the directories are sorted by LogViewPlus. Are they sorted by date modified? If yes, then that + multiple filenames + wildcard directory matching would be the solution.

RE: Watch the most recent folder

LogViewPlus Support — Sun, 24 Apr 2022 12:38:40 GMT

I can see where some users of this kind of directory structure might want to have distinct merge file names. I think we should aim to include this as well by defaulting to the name of the directory that matched by the wildcard.

I am not sure how this would be implemented as I do not want to further complicate the directory monitor settings (for most users anyway). It may be something like a reserved word.

I will have a think, but we will aim for this as well.

Toby

RE: Watch the most recent folder

LogViewPlus Support — Sun, 24 Apr 2022 12:16:00 GMT

That helps a lot Samaursa - thanks for the clarification.

If we support directory wildcard matching and multiple file names - I think this could meet your requirements. You could define your monitor as:

[code]/root/MyApp*
log1.txt|*.log[/code]
If the application is stopped and restarted, a new directory would be created which the directory monitor should pick up. At that point, the log files will be opened and added to the same merge file. There is no way to define a dynamic merge file name.

I think these would be really valuable enhancements to LogViewPlus. It is a bit late in this release cycle, but this is something we can look at in the next release.

Thanks again,

Toby

RE: Watch the most recent folder

samaursa — Sat, 23 Apr 2022 22:19:08 GMT

I'll try to recreate the example to explain a bit more

The log files are generated with each run of multiple applications. All applications dump their log files nested under the same root folder (let's call it LofFiles). The nested folder starts with the name of the application followed by a time stamp. This means that multiple runs of the apps will create multiple folders even if they are a few minutes apart. See below for an example.

All applications that dump their logs under this same LogFiles folder use the same custom logging framework. This logging framework writes the log files like this:

So, MyMainApp will have a warnings.txt and trace.txt and so will SomeOtherApp. The above files (most of them at least) will be found under all the folders in the first screenshot. Which means that ultimately, the warnings.txt and trace.txt files (and more) are ambiguous. That is, without knowing the full path, it is not possible to differentiate between a warnings.txt from one app vs another.

In addition to that, there are some massive log files that I want to ignore completely. For example, in the above screenshot, these would be CRC.txt, CRC2.txt and CRC3.txt. There are other log files I would like to ignore as well.

Now, coming to the directory watcher, I currently have no way to...
- ...ignore SomeOtherApp folder (there could be SomeOtherApp2 as well and so on)
- ...have the watcher pick the last modified folder of MyMainApp. If I don't do that, then multiple warnings.txt (and others) show up which I don't want to see
- ...no way to include multiple file names in the same search. Although maybe the regex filter could work? I am not great at regex and have not tried that root yet. It would be nice to have the file filter of the directory watcher accept multiple file filters with a comma or semicolon e.g. warnings.txt;dbtrace.txt;*.log
- ...ignore the CRC (and other) files while dragging and dropping

Because of the way the logging is setup, ultimately, I want from the latest MyMainApp folder, all the relevant log files (warnings.txt, dbtrace.txt, ai.txt) opened and merged into a single log file, which is then opened up.

I hope this clears it up? Please let me know if it is still unclear

[quote]What if we supported wildcard names in the directory path? Would that help?[/quote]

Adding wildcard support to the directory path would certainly help with filtering out the unwanted app folders.

RE: Watch the most recent folder

LogViewPlus Support — Sat, 23 Apr 2022 07:34:23 GMT

Hi Samaursa,

This is the official way to request new features, but it is not yet clear to me the best way to meet your requirements.

The directory monitor feature needs to be kept as simple as possible. Adding a date field and an 'ignore' field would push the number of core configuration options from 3 to 5. I am not sure this is the right thing to do.

One clarification is that the "ignore time frame" check works on the last modified date as reported by the operating system. Are these files local or remote? It sounds like you have some files that have not been written to in awhile, but you still want to open them with the same directory monitor - is this correct? I ask because I wonder if the last write time check is not working correctly in your case.

It is still not clear to me why the existing options are not working for you. It sounds like you have a lot of subdirectories and they are all receiving log updates. You only care about 1 subdirectory - the latest. The name of this directory cannot be known in advance. Is this correct?

What if we supported wildcard names in the directory path? Would that help?

Also, what would you expect the directory monitor to do if it was monitoring the latest subdirectory and a new latest subdirectory was created?

Thanks,

Toby

RE: Watch the most recent folder

samaursa — Sat, 23 Apr 2022 01:49:11 GMT

[quote]The root directory. It sounds like this needs to be the parent directory. You can recurse the parent to find log files in sub directories.[/quote]

Right, however what I would like is to open only the latest folder in the parent folder. Because all folders have the same files with the same names, if I use a log file pattern and use the root folder, then it will match several log files of the same name. If I use a time frame where I ignore files more than an hour ago, then I may miss the log of a long running application (which is quite often).

It would be nice if in addition to the time frame filter, there's a 'last modified' filter so that I can pick the most recently modified log file (which should essentially be the latest).

Since there is a fuzzy file search, it would be great if the directory filter itself was also fuzzy. This way, I could filter some folders. In our case, the same root folder has logs for `applicationA_xxxxxx` and `applicationB_xxxxxx` and I would like the filter to only find files from `applicationA` folders.

[quote]LogViewPlus does not have an 'exclude' file feature on directory monitors.[/quote]

Is there an official way to request new features? This would be very useful to have.

RE: Watch the most recent folder

LogViewPlus Support — Fri, 22 Apr 2022 20:20:08 GMT

Hi Samaursa,

It's not clear where you are running into difficulties, but I can try to offer some suggestions.

There are three main elements to configuring a directory monitor:
1. The root directory. It sounds like this needs to be the parent directory. You can recurse the parent to find log files in sub directories.
2. The log file name pattern. It looks like the log names above have an app name followed by a date. So you could use a wildcard like "appname*.csv" to match any date.
3. The time frame. From your description, I would suggest ignoring any files created more than an hour ago.

You need to apply some combination of the above three constraints to open your target files. LogViewPlus does not have an 'exclude' file feature on directory monitors.

It might also be worth experimenting with multiple directory monitors - each with a specific job.

Hope that helps,

Toby