Welcome! Log In Create A New Profile

Advanced

Re: Workers CPU leak [epoll_wait,epoll_ctl]

Maxim Dounin
March 09, 2016 10:14AM
Hello!

On Wed, Mar 09, 2016 at 09:28:20AM -0500, vizl wrote:

> Debug log regarding to hanged PID 7479 http://dev.vizl.org/debug.log.txt

This looks like a threads + sendfile() loop due to non-atomic updates of
underlying file, similar to one recently reported on the Russian
mailing list.

Correct solution would be to fix your system to update files
atomically instead of overwriting them in place.

The patch below will resolve CPU hog and will log an alert
instead:

# HG changeset patch
# User Maxim Dounin <mdounin@mdounin.ru>
# Date 1457536139 -10800
# Wed Mar 09 18:08:59 2016 +0300
# Node ID e96e5dfe4ff8ffe301264c3eb2771596fae24d38
# Parent 93049710cb7f6ea91fa9bd707e88fbe79d82d0ef
Truncation detection in sendfile() on Linux.

This addresses connection hangs as observed in ticket #504, and
CPU hogs with "aio threads; sendfile on" as reported in the mailing list,
see http://mailman.nginx.org/pipermail/nginx-ru/2016-March/057638.html.

The alert is identical to one used on FreeBSD.

diff --git a/src/os/unix/ngx_linux_sendfile_chain.c b/src/os/unix/ngx_linux_sendfile_chain.c
--- a/src/os/unix/ngx_linux_sendfile_chain.c
+++ b/src/os/unix/ngx_linux_sendfile_chain.c
@@ -292,6 +292,19 @@ eintr:
}
}

+ if (n == 0) {
+ /*
+ * if sendfile returns zero, then someone has truncated the file,
+ * so the offset became beyond the end of the file
+ */
+
+ ngx_log_error(NGX_LOG_ALERT, c->log, 0,
+ "sendfile() reported that \"%s\" was truncated at %O",
+ file->file->name.data, file->file_pos);
+
+ return NGX_ERROR;
+ }
+
ngx_log_debug3(NGX_LOG_DEBUG_EVENT, c->log, 0, "sendfile: %z of %uz @%O",
n, size, file->file_pos);

@@ -349,6 +362,19 @@ ngx_linux_sendfile_thread(ngx_connection
return NGX_ERROR;
}

+ if (ctx->err != NGX_AGAIN && ctx->sent == 0) {
+ /*
+ * if sendfile returns zero, then someone has truncated the file,
+ * so the offset became beyond the end of the file
+ */
+
+ ngx_log_error(NGX_LOG_ALERT, c->log, 0,
+ "sendfile() reported that \"%s\" was truncated at %O",
+ file->file->name.data, file->file_pos);
+
+ return NGX_ERROR;
+ }
+
*sent = ctx->sent;

return (ctx->sent == ctx->size) ? NGX_DONE : NGX_AGAIN;

--
Maxim Dounin
http://nginx.org/

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Subject Author Posted

Workers CPU leak [epoll_wait,epoll_ctl]

vizl February 24, 2016 09:17AM

Re: Workers CPU leak [epoll_wait,epoll_ctl]

vizl February 24, 2016 09:39AM

Re: Workers CPU leak [epoll_wait,epoll_ctl]

Valentin V. Bartenev February 24, 2016 10:26AM

Re: Workers CPU leak [epoll_wait,epoll_ctl]

vizl March 04, 2016 07:12AM

Re: Workers CPU leak [epoll_wait,epoll_ctl]

Maxim Konovalov March 04, 2016 07:26AM

Re: Workers CPU leak [epoll_wait,epoll_ctl]

Valentin V. Bartenev March 04, 2016 07:32AM

Re: Workers CPU leak [epoll_wait,epoll_ctl]

vizl March 04, 2016 08:25AM

Re: Workers CPU leak [epoll_wait,epoll_ctl]

Valentin V. Bartenev March 04, 2016 08:46AM

Re: Workers CPU leak [epoll_wait,epoll_ctl]

vizl March 04, 2016 09:42AM

Re: Workers CPU leak [epoll_wait,epoll_ctl]

Valentin V. Bartenev March 04, 2016 09:46AM

Re: Workers CPU leak [epoll_wait,epoll_ctl]

vizl March 09, 2016 09:28AM

Re: Workers CPU leak [epoll_wait,epoll_ctl]

Maxim Dounin March 09, 2016 10:14AM

Re: Workers CPU leak [epoll_wait,epoll_ctl]

vizl March 09, 2016 10:56AM

Re: Workers CPU leak [epoll_wait,epoll_ctl]

Maxim Dounin March 09, 2016 11:20AM

Re: Workers CPU leak [epoll_wait,epoll_ctl]

vizl March 10, 2016 09:08AM

Re: Workers CPU leak [epoll_wait,epoll_ctl]

vizl March 20, 2016 04:34PM

Re: Workers CPU leak [epoll_wait,epoll_ctl]

Maxim Konovalov March 21, 2016 04:10AM

Re: Workers CPU leak [epoll_wait,epoll_ctl]

vizl March 21, 2016 08:00AM

Re: Workers CPU leak [epoll_wait,epoll_ctl]

Maxim Dounin March 21, 2016 09:30AM



Sorry, only registered users may post in this forum.

Click here to login

Online Users

Guests: 102
Record Number of Users: 6 on February 13, 2018
Record Number of Guests: 421 on December 02, 2018
Powered by nginx      Powered by FreeBSD      PHP Powered      Powered by MariaDB      ipv6 ready