Hello!

On Fri, Apr 10, 2020 at 11:26:17AM -0400, patademahesh wrote:

> We are using nginx as reverse proxy to cache static content for a moodle LMS
> site. The caching part is working fine but we started facing the cache path
> disk full issues. When we checked using du, it was reporting very low used
> space.Then we checked lsof output we found too many deleted file entries. We
> realized, that file gets deleted but it holds the FD and disk space is never
> released until we restart the nginx server. Most of the files were from
> proxy_temp_path location. We tried this with tmpfs and normal disk but the
> end result was same.
>
> # nginx -v
> nginx version: nginx/1.10.3 (Ubuntu)

Note that when writing temporary files when proxying (without
caching, but there is no caching configured in "location /"), it
is quite normal that temporary files are unlinked (deleted) right
after creation and cannot be seen by "du". This way temporary
files are automatically removed by the system as long as a file is
closed - or if nginx is killed or crashes.

If you think that temporary files are not removed from disk even
if corresponding client connections are closed - this might
indicate a socket leak. Usually, socket leaks can be seen by
other symptoms as well - such as connections in the CLOSED state
as shown by "netstat -an", or "open socket ... left in connection ..."
alerts during graceful shutdown of worker processes. Given
that you are using an ancient nginx version, socket leaks might be
the case - there are at least some fixed since 1.10.3, mostly in
HTTP/2.

On the other hand, if deleted files is the only symptom you are
seeing, it might simply indicate that corresponding client
connection is still open and the client is slowly downloading the
response in question. If that's true, you may want to either use
a filesystem with more space for temporary files, or tune
proxy buffering, see here:

http://nginx.org/r/proxy_max_temp_file_size

--
Maxim Dounin
http://mdounin.ru/
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx