Welcome! Log In Create A New Profile

Advanced

Add support for buffering is scripted logs

August 07, 2017 06:38AM
Hi all,

The attached patch adds support for log buffering when using variables in the access log file name.

The use case is this - we use nginx to receive analytics beacons and write them to the access log.
We'd like to have a log file per hour that contains the logs of the specific hour. If we use some external
script to perform log rotate, we cannot avoid log lines slipping between adjacent files.
It's important for us to have the log lines partitioning accurate, in case we need to reindex a specific
hour to the database.
Scripted logs seem perfect for this, but we don't want to give up on log compression, since the
number of events can be high.

The patch relies on the open file cache to keep the context (buffer + flush event) of each file.
In order to do that, I added the ability to register callbacks in open file cache for these events:
1. init - new cached file object created
2. flush - the file handle of a cached file is closed
3. free - the cached file is destroyed
The context is allocated right after the ngx_cached_open_file_t struct, this was done in order to avoid
increasing the size of the ngx_cached_open_file_t struct (compared to the alternative of adding some
opaque pointer on this struct).
Therefore, there are no memory implications for "regular" (log = 0) open file caches, the overhead is
only an extra 'if' on the creation / deletion of the cached file object (no impact at all on cache hit,
which is probably the most performance sensitive)

Btw, on a similar subject - can anyone explain the purpose of checking the existence of the root dir
in scripted logs? Only explanation I could think of is to provide some security protection in case $uri
is used in the script, but that sounds like a very specific use case to me....
Anyway, if that is indeed the case, maybe it makes sense to add conf directive to enable/disable
this behavior?

Thank you!

Eran

_______________________________________________
nginx-devel mailing list
nginx-devel@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx-devel
Subject Author Views Posted

Add support for buffering is scripted logs Attachments

erankor 868 August 07, 2017 06:38AM

Re: Add support for buffering is scripted logs

Maxim Dounin 441 August 14, 2017 11:46AM

RE: Add support for buffering is scripted logs

erankor 704 August 14, 2017 12:02PM

Re: Add support for buffering is scripted logs

Maxim Dounin 425 August 14, 2017 12:56PM

RE: Add support for buffering is scripted logs

erankor 665 August 14, 2017 01:12PM

Re: Add support for buffering is scripted logs

Maxim Dounin 436 August 14, 2017 01:36PM

RE: Add support for buffering is scripted logs

erankor 669 August 14, 2017 02:02PM

Re: Add support for buffering is scripted logs

Alexey Ivanov 423 August 14, 2017 02:26PM

RE: Add support for buffering is scripted logs

erankor 660 August 14, 2017 05:26PM

Re: Add support for buffering is scripted logs

Alexey Ivanov 474 August 14, 2017 06:44PM

Re: Add support for buffering is scripted logs

Maxim Dounin 442 August 14, 2017 03:02PM



Sorry, you do not have permission to post/reply in this forum.

Online Users

Guests: 331
Record Number of Users: 8 on April 13, 2023
Record Number of Guests: 500 on July 15, 2024
Powered by nginx      Powered by FreeBSD      PHP Powered      Powered by MariaDB      ipv6 ready