Hello!

On Wed, Mar 09, 2016 at 09:28:20AM -0500, vizl wrote:

> Debug log regarding to hanged PID 7479 http://dev.vizl.org/debug.log.txt

This looks like a threads + sendfile() loop due to non-atomic updates of
underlying file, similar to one recently reported on the Russian
mailing list.

Correct solution would be to fix your system to update files
atomically instead of overwriting them in place.

The patch below will resolve CPU hog and will log an alert
instead:

# HG changeset patch
# User Maxim Dounin <mdounin@mdounin.ru>
# Date 1457536139 -10800
# Wed Mar 09 18:08:59 2016 +0300
# Node ID e96e5dfe4ff8ffe301264c3eb2771596fae24d38
# Parent 93049710cb7f6ea91fa9bd707e88fbe79d82d0ef
Truncation detection in sendfile() on Linux.

This addresses connection hangs as observed in ticket #504, and
CPU hogs with "aio threads; sendfile on" as reported in the mailing list,
see http://mailman.nginx.org/pipermail/nginx-ru/2016-March/057638.html.

The alert is identical to one used on FreeBSD.

diff --git a/src/os/unix/ngx_linux_sendfile_chain.c b/src/os/unix/ngx_linux_sendfile_chain.c
--- a/src/os/unix/ngx_linux_sendfile_chain.c
+++ b/src/os/unix/ngx_linux_sendfile_chain.c
@@ -292,6 +292,19 @@ eintr:
}
}

+ if (n == 0) {
+ /*
+ * if sendfile returns zero, then someone has truncated the file,
+ * so the offset became beyond the end of the file
+ */
+
+ ngx_log_error(NGX_LOG_ALERT, c->log, 0,
+ "sendfile() reported that \"%s\" was truncated at %O",
+ file->file->name.data, file->file_pos);
+
+ return NGX_ERROR;
+ }
+
ngx_log_debug3(NGX_LOG_DEBUG_EVENT, c->log, 0, "sendfile: %z of %uz @%O",
n, size, file->file_pos);

@@ -349,6 +362,19 @@ ngx_linux_sendfile_thread(ngx_connection
return NGX_ERROR;
}

+ if (ctx->err != NGX_AGAIN && ctx->sent == 0) {
+ /*
+ * if sendfile returns zero, then someone has truncated the file,
+ * so the offset became beyond the end of the file
+ */
+
+ ngx_log_error(NGX_LOG_ALERT, c->log, 0,
+ "sendfile() reported that \"%s\" was truncated at %O",
+ file->file->name.data, file->file_pos);
+
+ return NGX_ERROR;
+ }
+
*sent = ctx->sent;

return (ctx->sent == ctx->size) ? NGX_DONE : NGX_AGAIN;

--
Maxim Dounin
http://nginx.org/

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx