April 13, 2016 05:08AM
Hello again.

Maxim Dounin Wrote:
-------------------------------------------------------
> Hello!
>
> On Thu, Feb 25, 2016 at 05:46:32AM -0500, vergil wrote:
>
> > vedranf Wrote:
> > -------------------------------------------------------
> > > Hello,
> > >
> > > I'm having an issue where nginx (1.8) cache manager suddenly just
> > > stops deleting content thus the disk soon ends up being full until
> I
> > > restart it by hand. After it is restarted, it works normally for a
> > > couple of days, but then it happens again. Cache has some 30-40k
> > > files, nothing huge. Relevant config lines are:
>
> [...]
>
> > We have the same problem, but i'm not sure, that this is caused by
> often
> > nginx restarts.
>
> This particular case was traced to segmentation faults, likely
> caused by 3rd party modules.
>
> [...]
>
> > Also, i think it's somehow related to write connection leak. (see
> image
> > link)
> >
> >
> https://s3.eu-central-1.amazonaws.com/drive-public-eu/nginx/betelgeuse
> _nginx_connections.PNG
>
> [...]
>
> > As you see write connections continuously grows. (When we had to
> power off
> > the machine it's reached ~60k).
> >
> > For counting nginx connections we use standard
> http_stub_status_module.
> >
> > I think that nginx "reference counter" could be broken, because
> total
> > established TCP connection remains the same all the time.
>
> Writing connections will grow due to segmentation faults as well,
> so you are likely have the same problem. See basic
> recommendations in my initial answer in this threads.

I've maded custom nginx build using latest version (1.9.13) without 3rd party modules:

nginx -V
nginx version: nginx/1.9.13
built by gcc 4.8.4 (Ubuntu 4.8.4-2ubuntu1~14.04.1)
built with OpenSSL 1.0.1f 6 Jan 2014
TLS SNI support enabled
configure arguments: --with-cc-opt='-g -O2 -fPIE -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2' --with-ld-opt='-Wl,-Bsymbolic-functions -fPIE -pie -Wl,-z,relro -Wl,-z,now' --prefix=/usr/share/nginx --conf-path=/etc/nginx/nginx.conf --http-log-path=/var/log/nginx/access.log --error-log-path=/var/log/nginx/error.log --lock-path=/var/lock/nginx.lock --pid-path=/run/nginx.pid --http-client-body-temp-path=/var/lib/nginx/body --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-proxy-temp-path=/var/lib/nginx/proxy --http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-debug --with-pcre-jit --with-ipv6 --with-http_ssl_module --with-http_stub_status_module --with-http_realip_module --with-http_auth_request_module --with-http_addition_module --with-http_dav_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_v2_module --with-http_sub_module --with-http_xslt_module --with-stream --with-stream_ssl_module --with-mail --with-mail_ssl_module --with-threads

Nothing changed: connections continuously grow, cache manager works fine and not filled the disk yet, but i think it's a matter of 2-3 days.

PIDs didn't changed since the start, and log didn't contain "worker process exited ..." messages.

Regards,
Alexander.
Subject Author Posted

Cache manager occasionally stops deleting cached files

vedranf February 18, 2016 11:20AM

Re: Cache manager occasionally stops deleting cached files

Maxim Dounin February 18, 2016 12:12PM

Re: Cache manager occasionally stops deleting cached files

vedranf February 19, 2016 08:00AM

Re: Cache manager occasionally stops deleting cached files

Maxim Dounin February 19, 2016 08:30AM

Re: Cache manager occasionally stops deleting cached files

vergil February 25, 2016 05:46AM

Re: Cache manager occasionally stops deleting cached files

Maxim Dounin February 25, 2016 09:22AM

Re: Cache manager occasionally stops deleting cached files

vergil April 13, 2016 05:08AM



Sorry, only registered users may post in this forum.

Click here to login

Online Users

Guests: 258
Record Number of Users: 8 on April 13, 2023
Record Number of Guests: 500 on July 15, 2024
Powered by nginx      Powered by FreeBSD      PHP Powered      Powered by MariaDB      ipv6 ready