Welcome! Log In Create A New Profile

Advanced

Re: Workers CPU leak [epoll_wait,epoll_ctl]

Valentin V. Bartenev
February 24, 2016 10:26AM
On Wednesday 24 February 2016 09:17:01 vizl wrote:
> Hello, I have strange issuses with nginx workers. For some time after start
> Nginx I notice that some process of workers cause high load to CPU (
> principally sys CPU).
>
> At first I've got syscall traces from one of such process:
>
> futex(0x157d914, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x157d910, {FUTEX_OP_SET, 0,
> FUTEX_OP_CMP_GT, 1}) = 1
> epoll_wait(38, {{EPOLLIN, {u32=7156096, u64=7156096}}}, 512, -1) = 1
> epoll_ctl(38, EPOLL_CTL_ADD, 178, {EPOLLOUT|EPOLLET, {u32=3888102096,
> u64=140028411886288}}) = 0
> epoll_wait(38, {{EPOLLOUT, {u32=3888102096, u64=140028411886288}}}, 512, -1)
> = 1
> epoll_ctl(38, EPOLL_CTL_DEL, 178, 7ffda2bc7f30) = 0
> futex(0x157d914, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x157d910, {FUTEX_OP_SET, 0,
> FUTEX_OP_CMP_GT, 1}) = 1
> epoll_wait(38, {{EPOLLIN, {u32=7156096, u64=7156096}}}, 512, -1) = 1
> epoll_ctl(38, EPOLL_CTL_ADD, 178, {EPOLLOUT|EPOLLET, {u32=3888102096,
> u64=140028411886288}}) = 0
> epoll_wait(38, {{EPOLLOUT, {u32=3888102096, u64=140028411886288}}}, 512, -1)
> = 1
> epoll_ctl(38, EPOLL_CTL_DEL, 178, 7ffda2bc7f30) = 0
> futex(0x157d914, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x157d910, {FUTEX_OP_SET, 0,
> FUTEX_OP_CMP_GT, 1}) = 1
> futex(0x157d8d0, FUTEX_WAKE_PRIVATE, 1) = 1
>
> epoll_wait, epoll_ctl, futex are repeated circularly.
>
> Then I've got lsof of process and see who owns of 38 file descriptor:
>
> nginx 18862 www 38u a_inode 0,9 0 6
> [eventpoll]
>
> also I see several CLOSE_WAIT sockets
>
> nginx 18862 www 101u IPv4 85643376 0t0 TCP
> 154.59.82.194:http->105.107.179.210:24519 (CLOSE_WAIT)
> nginx 18862 www 133r REG 8,3 0 4743
> /mnt/ssd1/wwwroot/71/7/27394667.mp4 (deleted)
> nginx 18862 www 178u IPv4 86054929 0t0 TCP
> 154.59.82.194:http->5adc98ed.bb.sky.com:45665 (CLOSE_WAIT)
> nginx 18862 www 179r REG 8,3 0 5098
> /mnt/ssd1/wwwroot/21/9/29603499.mp4 (deleted)
>
>
> Nginx has such version and modules:
>
> nginx version: nginx/1.9.11
> built with OpenSSL 1.0.2f 28 Jan 2016
> TLS SNI support enabled
> configure arguments: --prefix=/usr --conf-path=/etc/nginx/nginx.conf
> --error-log-path=/var/log/nginx/error_log --pid-path=/run/nginx.pid
> --lock-path=/run/lock/nginx.lock --with-cc-opt=-I/usr/include
> --with-ld-opt=-L/usr/lib64 --http-log-path=/var/log/nginx/access_log
> --http-client-body-temp-path=/var/lib/nginx/tmp/client
> --http-proxy-temp-path=/var/lib/nginx/tmp/proxy
> --http-fastcgi-temp-path=/var/lib/nginx/tmp/fastcgi
> --http-scgi-temp-path=/var/lib/nginx/tmp/scgi
> --http-uwsgi-temp-path=/var/lib/nginx/tmp/uwsgi --with-file-aio --with-ipv6
> --with-pcre --with-threads --without-http_autoindex_module
> --without-http_fastcgi_module --without-http_geo_module
> --without-http_limit_req_module --without-http_limit_conn_module
> --without-http_memcached_module --without-http_uwsgi_module
> --with-http_flv_module --with-http_gzip_static_module --with-http_mp4_module
> --with-http_perl_module
> --add-module=external_module/headers-more-nginx-module-0.261
> --add-module=external_module/ngx_estreaming_module-0.01
> --add-module=external_module/ngx_slice_module-0.01 --with-http_ssl_module
> --without-mail_imap_module --without-mail_pop3_module
> --without-mail_smtp_module --user='www --group=www'
>
> and using for video streaming.
>
> Has anyone encountered such behavior ? Help please.
>
[..]

Could you provide a minimal configuration that is causing problems
with debug log? See: http://nginx.org/en/docs/debugging_log.html

wbr, Valentin V. Bartenev

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Subject Author Posted

Workers CPU leak [epoll_wait,epoll_ctl]

vizl February 24, 2016 09:17AM

Re: Workers CPU leak [epoll_wait,epoll_ctl]

vizl February 24, 2016 09:39AM

Re: Workers CPU leak [epoll_wait,epoll_ctl]

Valentin V. Bartenev February 24, 2016 10:26AM

Re: Workers CPU leak [epoll_wait,epoll_ctl]

vizl March 04, 2016 07:12AM

Re: Workers CPU leak [epoll_wait,epoll_ctl]

Maxim Konovalov March 04, 2016 07:26AM

Re: Workers CPU leak [epoll_wait,epoll_ctl]

Valentin V. Bartenev March 04, 2016 07:32AM

Re: Workers CPU leak [epoll_wait,epoll_ctl]

vizl March 04, 2016 08:25AM

Re: Workers CPU leak [epoll_wait,epoll_ctl]

Valentin V. Bartenev March 04, 2016 08:46AM

Re: Workers CPU leak [epoll_wait,epoll_ctl]

vizl March 04, 2016 09:42AM

Re: Workers CPU leak [epoll_wait,epoll_ctl]

Valentin V. Bartenev March 04, 2016 09:46AM

Re: Workers CPU leak [epoll_wait,epoll_ctl]

vizl March 09, 2016 09:28AM

Re: Workers CPU leak [epoll_wait,epoll_ctl]

Maxim Dounin March 09, 2016 10:14AM

Re: Workers CPU leak [epoll_wait,epoll_ctl]

vizl March 09, 2016 10:56AM

Re: Workers CPU leak [epoll_wait,epoll_ctl]

Maxim Dounin March 09, 2016 11:20AM

Re: Workers CPU leak [epoll_wait,epoll_ctl]

vizl March 10, 2016 09:08AM

Re: Workers CPU leak [epoll_wait,epoll_ctl]

vizl March 20, 2016 04:34PM

Re: Workers CPU leak [epoll_wait,epoll_ctl]

Maxim Konovalov March 21, 2016 04:10AM

Re: Workers CPU leak [epoll_wait,epoll_ctl]

vizl March 21, 2016 08:00AM

Re: Workers CPU leak [epoll_wait,epoll_ctl]

Maxim Dounin March 21, 2016 09:30AM



Sorry, only registered users may post in this forum.

Click here to login

Online Users

Guests: 100
Record Number of Users: 6 on February 13, 2018
Record Number of Guests: 421 on December 02, 2018
Powered by nginx      Powered by FreeBSD      PHP Powered      Powered by MariaDB      ipv6 ready