Welcome! Log In Create A New Profile

Advanced

Per-Request Unique Identifier (Log Correlation)

November 05, 2010 02:31AM
There is information available to log by the fastcgi backend which is not available to nginx, and vice versa (e.g., $body_bytes_sent). I need some way to match these two logs together after the fact. This necessarily implies a quasi-GUID guaranteed to be unique to each request. Preferably it should be something generated by nginx and fed to backend through a fastcgi_param (or for those using proxy, through a request header).

One idea is to combine these:

(a) nginx worker $pid
(b) a timestamp (<=1s res.)
(c) a request counter internal to each nginx worker process.

Together, these *should* together fulfill all the foregoing requirements in real-world practice. (Information leakage is not a concern, since this will only be used internally. Efficiency is obviously paramount.) However as far as I can tell, there is no such thing as (c), nor anything else which can be used to uniquely identify each request.

(Yes there is other stuff which should be added to *theoretically* guarantee uniqueness. I am aware of the difficulty of finding a portable high-performance solution for that, e.g. from maildir discussions.)

Any chance to get a monotonically increasing counter of requests since worker process start? I am guessing this would take about 6 lines of code; but like the old legend about Tesla, the trick is knowing where to put those 6 lines! I expect many people could use this to sync logs across nginx frontend and fastcgi or proxy backend.

Performance impact of initializing unsigned long long (>= 64-bit) counter on worker start, ++ on each request, and keeping 8 more bytes in memory per process is negligible to both cycles and processor cache usage (unless there are any internal thread issues I don't know about). I don't know the impact of exposing that as a $worker_request_counter variable; but I assume nginx handles config variables with very high efficiency. For a worker serving 1,000 req./s, uint64_t or equivalent would overflow after about 584 million years; I think most people reload their nginx workers more often than that.

Obviously all of the above is kind of a kludge, and any theoretically elegant solution would be best.

Thanks,

helen
Subject Author Posted

Per-Request Unique Identifier (Log Correlation)

helen November 05, 2010 02:31AM



Sorry, only registered users may post in this forum.

Click here to login

Online Users

Guests: 255
Record Number of Users: 8 on April 13, 2023
Record Number of Guests: 421 on December 02, 2018
Powered by nginx      Powered by FreeBSD      PHP Powered      Powered by MariaDB      ipv6 ready