Hello everyone.

I have some problem about subrequest.

My module during the processing of one request,
handler() in the access phase issue sequential sub requests sequentially,
while dynamically obtain data from the file cache or upstream,
ngx_http_output_filter calling it many times,
send data permanently back to the client little by little.

This seems to work at first glance, but it has the problem of dealing with memory, file descriptors, and disk resources until it processes the request and ends the connection with the client.


1. Leaking memory

Since nginx does not release the memory reserved by r->pool until processing of one request is completed.
In such processing as returning a response permanently, the memory usage will continue to increase.

In order to solve this problem, I tried to purge r->out and r->cache->buf, etc buffers by ngx_pfree() in subrequest finalize_handler, but reserved memory by ngx_palloc_small() could not purge.

In my module, the total memory allocated by ngx_palloc_small() in one subrequest is now more than 4 MB without r->out and r->cache->buf.

I think that I want to be able to reserve specify another new pool when creating a subrequest, but since nginx shares the pool with the mainrequest and subrequests.
It is difficult to purge separately only the memories that have been created by subrequest.

Is there a good solution about this problem?
Can I control it to samehow by cleanup handler?


2. Increase of fd and disk

nginx write temporary file to store data to get from upstream etc, and rename to standard hash name.

But fd of there temorary files is not closed() during processing of this request, the same file name attempting to rename to a file consumes both old and new fd and disk.

$ ls -l /proc/nginx-worker-pid/fd

21 /var/cache/nginx/temp-hash-file (delete)
22 /var/cache/nginx/temp-hash-file (delete)
23 /var/cache/nginx/temp-hash-file
24 /var/cache/nginx/temp-hash-file (delete)
:
:

Without caching, this kind of thing did not happen.

As long as you keep repeating caching sub requests, you can not prevent disk growth.

I close the following fd in filnalize_handler of subrequest,
It seems to have been solved, but this attempt is correct could it be?

ngx_close(r->upstream->pipe->temp_file.file.fd);
ngx_close(r->cahce->file.fd);
r->upstream->pipe->temp_file.file.fd = NGX_INVALID_FILE;
r->cache->file.fd = NGX_INVALID_FILE;

Michihiro