Welcome! Log In Create A New Profile

Advanced

Re: Too many deleted open files in proxy_temp_path

Maxim Dounin
April 12, 2020 10:50AM
Hello!

On Fri, Apr 10, 2020 at 11:26:17AM -0400, patademahesh wrote:

> We are using nginx as reverse proxy to cache static content for a moodle LMS
> site. The caching part is working fine but we started facing the cache path
> disk full issues. When we checked using du, it was reporting very low used
> space.Then we checked lsof output we found too many deleted file entries. We
> realized, that file gets deleted but it holds the FD and disk space is never
> released until we restart the nginx server. Most of the files were from
> proxy_temp_path location. We tried this with tmpfs and normal disk but the
> end result was same.
>
> # nginx -v
> nginx version: nginx/1.10.3 (Ubuntu)

Note that when writing temporary files when proxying (without
caching, but there is no caching configured in "location /"), it
is quite normal that temporary files are unlinked (deleted) right
after creation and cannot be seen by "du". This way temporary
files are automatically removed by the system as long as a file is
closed - or if nginx is killed or crashes.

If you think that temporary files are not removed from disk even
if corresponding client connections are closed - this might
indicate a socket leak. Usually, socket leaks can be seen by
other symptoms as well - such as connections in the CLOSED state
as shown by "netstat -an", or "open socket ... left in connection ..."
alerts during graceful shutdown of worker processes. Given
that you are using an ancient nginx version, socket leaks might be
the case - there are at least some fixed since 1.10.3, mostly in
HTTP/2.

On the other hand, if deleted files is the only symptom you are
seeing, it might simply indicate that corresponding client
connection is still open and the client is slowly downloading the
response in question. If that's true, you may want to either use
a filesystem with more space for temporary files, or tune
proxy buffering, see here:

http://nginx.org/r/proxy_max_temp_file_size

--
Maxim Dounin
http://mdounin.ru/
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Subject Author Posted

Too many deleted open files in proxy_temp_path

patademahesh April 10, 2020 11:26AM

Re: Too many deleted open files in proxy_temp_path

J.R. April 10, 2020 11:20PM

Re: Too many deleted open files in proxy_temp_path

patademahesh April 11, 2020 01:01AM

Re: Too many deleted open files in proxy_temp_path

Maxim Dounin April 12, 2020 10:50AM

Re: Too many deleted open files in proxy_temp_path

renoirb April 12, 2020 11:39PM



Sorry, only registered users may post in this forum.

Click here to login

Online Users

Guests: 63
Record Number of Users: 6 on February 13, 2018
Record Number of Guests: 421 on December 02, 2018
Powered by nginx      Powered by FreeBSD      PHP Powered      Powered by MariaDB      ipv6 ready