Hello!

On Tue, Nov 23, 2021 at 10:44:00AM +0300, Sergey Kandaurov wrote:

> > On 23 Nov 2021, at 06:12, Maxim Dounin <mdounin@mdounin.ru> wrote:
> >
> > On Thu, Nov 18, 2021 at 07:46:48PM +0300, Sergey Kandaurov wrote:
> >
> >>> On 16 Nov 2021, at 17:41, Maxim Dounin <mdounin@mdounin.ru> wrote:
> >>>
> >>> On Tue, Nov 16, 2021 at 02:59:46PM +0300, Sergey Kandaurov wrote:
> >>>
> >>>>> On 11 Nov 2021, at 06:10, Maxim Dounin <mdounin@mdounin.ru> wrote:
> >>>>>
> >>>>> # HG changeset patch
> >>>>> # User Maxim Dounin <mdounin@mdounin.ru>
> >>>>> # Date 1636599377 -10800
> >>>>> # Thu Nov 11 05:56:17 2021 +0300
> >>>>> # Node ID 76e072a6947a221868705c13973de15319c0d921
> >>>>> # Parent 82b750b20c5205d685e59031247fe898f011394e
> >>>>> HTTP/2: fixed sendfile() aio handling.
> >>>>>
> >>>>> With sendfile() in threads ("aio threads; sendfile on;"), client connection
> >>>>> can block on writing, waiting for sendfile() to complete. In HTTP/2 this
> >>>>> might result in the request hang, since an attempt to continue processig
> >>>>
> >>>> processing
> >>>
> >>> Fixed, thnx.
> >>>
> >>>>> in thread event handler will call request's write event handler, which
> >>>>> is usually stopped by ngx_http_v2_send_chain(): it does nothing if there
> >>>>> are no additional data and stream->queued is set. Further, HTTP/2 resets
> >>>>> stream's c->write->ready to 0 if writing blocks, so just fixing
> >>>>> ngx_http_v2_send_chain() is not enough.
> >>>>>
> >>>>> Can be reproduced with test suite on Linux with:
> >>>>>
> >>>>> TEST_NGINX_GLOBALS_HTTP="aio threads; sendfile on;" prove h2*.t
> >>>>>
> >>>>> The following tests currently fail: h2_keepalive.t, h2_priority.t,
> >>>>> h2_proxy_max_temp_file_size.t, h2.t, h2_trailers.t.
> >>>>>
> >>>>> Similarly, sendfile() with AIO preloading on FreeBSD can block as well,
> >>>>> with similar results. This is, however, harder to reproduce, especially
> >>>>> on modern FreeBSD systems, since sendfile() usually do not return EBUSY.
> >>>>
> >>>> does not
> >>>
> >>> Fixed, thnx.
> >>>
> >>>>> Fix is to post a write event on HTTP/2 connection in the thread event
> >>>>> handler (and aio preload handler). This ensures that sendfile() will be
> >>>>> completed and stream processing will be resumed by HTTP/2 code.
> >>>>>
> >>>>> diff --git a/src/http/ngx_http_copy_filter_module.c b/src/http/ngx_http_copy_filter_module.c
> >>>>> --- a/src/http/ngx_http_copy_filter_module.c
> >>>>> +++ b/src/http/ngx_http_copy_filter_module.c
> >>>>> @@ -250,6 +250,21 @@ ngx_http_copy_aio_sendfile_event_handler
> >>>>> r->aio = 0;
> >>>>> ev->complete = 0;
> >>>>>
> >>>>> +#if (NGX_HTTP_V2)
> >>>>> +
> >>>>> + if (r->stream) {
> >>>>> + /*
> >>>>> + * for HTTP/2, trigger a write event on the main connection
> >>>>> + * to handle sendfile() preload
> >>>>> + */
> >>>>> +
> >>>>> + ngx_post_event(r->stream->connection->connection->write,
> >>>>> + &ngx_posted_events);
> >>>>> + return;
> >>>>> + }
> >>>>> +
> >>>>> +#endif
> >>>>> +
> >>>>> r->connection->write->handler(r->connection->write);
> >>>>> }
> >>>>>
> >>>>> @@ -323,6 +338,20 @@ ngx_http_copy_thread_event_handler(ngx_e
> >>>>> r->main->blocked--;
> >>>>> r->aio = 0;
> >>>>>
> >>>>> +#if (NGX_HTTP_V2)
> >>>>> +
> >>>>> + if (r->stream) {
> >>>>> + /*
> >>>>> + * for HTTP/2, trigger a write event on the main connection
> >>>>> + * to handle sendfile() in threads
> >>>>> + */
> >>>>> +
> >>>>> + ngx_post_event(r->stream->connection->connection->write,
> >>>>> + &ngx_posted_events);
> >>>>> + }
> >>>>> +
> >>>>> +#endif
> >>>>> +
> >
> > [...]
> >
> > For the record, while testing this patch Sergey found another
> > issue with sendfile() in threads and HTTP/2: since HTTP/2 might
> > call sendfile() within the main connection, bypassing request
> > filter chain, normal r->aio flag checking to prevent multiple
> > operations do not work, and this eventually results in "task
> > already active" alerts due to duplicate operations being posted.
> > With the above patch this issue is much more likely to happen,
> > since it intentionally triggers write events on the main HTTP/2
> > connection.
> >
> > Below are two patches: the first one addresses the issue with
> > duplicate operations by additionally checking file->thread_task
> > before sendfile(), and the second one is a better alternative to
> > the above patch which doesn't post additional events on the main
> > connection.
> >
> > # HG changeset patch
> > # User Maxim Dounin <mdounin@mdounin.ru>
> > # Date 1637635671 -10800
> > # Tue Nov 23 05:47:51 2021 +0300
> > # Node ID 8a18b0bff1266db221fe35dc08f4483044ea0f86
> > # Parent 82b750b20c5205d685e59031247fe898f011394e
> > HTTP/2: fixed "task already active" with sendfile in threads.
> >
> > With sendfile in threads, "task already active" alerts might appear in logs
> > if a write event happens on the main HTTP/2 connection, triggering a sendfile
> > in threads while another thread operation is already running. Observed
> > with "aio threads; aio_write on; sendfile on;" and with thread event handlers
> > modified to post a write event to the main HTTP/2 connection (though can
> > happen without any modifications).
> >
> > Fix is to avoid starting a sendfile operation if file->thread_task indicates
> > that another thread operation is active.
> >
> > diff --git a/src/os/unix/ngx_linux_sendfile_chain.c b/src/os/unix/ngx_linux_sendfile_chain.c
> > --- a/src/os/unix/ngx_linux_sendfile_chain.c
> > +++ b/src/os/unix/ngx_linux_sendfile_chain.c
> > @@ -324,6 +324,18 @@ ngx_linux_sendfile_thread(ngx_connection
> > "linux sendfile thread: %d, %uz, %O",
> > file->file->fd, size, file->file_pos);
> >
> > + task = file->file->thread_task;
> > +
> > + if (task && task->event.active) {
> > + /*
> > + * with HTTP/2, another thread operation might be already running
> > + * if sendfile() is called as a result of a write event on the main
> > + * connection
> > + */
> > +
> > + return NGX_DONE;
> > + }
> > +
> > task = c->sendfile_task;
> >
> > if (task == NULL) {

After looking once again into it, I tend to think this patch is
incomplete. In particular, the particular check won't stop
additional sendfile() if there are multiple files and different
files use different thread tasks. While this is not something
possible with standard modules, but nevertheless.

Further, the same problem seems to apply to aio preloading (though
unlikely to happen in practice).

The following patch checks r->aio in relevant handlers to prevent
sendfile() when another operation is running:

# HG changeset patch
# User Maxim Dounin <mdounin@mdounin.ru>
# Date 1637723745 -10800
# Wed Nov 24 06:15:45 2021 +0300
# Node ID 3450841798597536d17ced29b35d1d90ce06ce0d
# Parent 3443c02ca1d183fe52bf8af66627c94be2b2f785
HTTP/2: fixed "task already active" with sendfile in threads.

With sendfile in threads, "task already active" alerts might appear in logs
if a write event happens on the main HTTP/2 connection, triggering a sendfile
in threads while another thread operation is already running. Observed
with "aio threads; aio_write on; sendfile on;" and with thread event handlers
modified to post a write event to the main HTTP/2 connection (though can
happen without any modifications).

Similarly, sendfile() with AIO preloading on FreeBSD can trigger duplicate
aio operation, resulting in "second aio post" alerts. This is, however,
harder to reproduce, especially on modern FreeBSD systems, since sendfile()
usually does not return EBUSY.

Fix is to avoid starting a sendfile operation if other thread operation
is active by checking r->aio in the thread handler (and, similarly, in
aio preload handler).

diff --git a/src/http/ngx_http_copy_filter_module.c b/src/http/ngx_http_copy_filter_module.c
--- a/src/http/ngx_http_copy_filter_module.c
+++ b/src/http/ngx_http_copy_filter_module.c
@@ -219,6 +219,22 @@ ngx_http_copy_aio_sendfile_preload(ngx_b
ngx_http_request_t *r;
ngx_output_chain_ctx_t *ctx;

+#if (NGX_HTTP_V2)
+
+ r = file->file->aio->data;
+
+ if (r->aio) {
+ /*
+ * with HTTP/2, another thread operation might be already running
+ * if sendfile() is called as a result of a write event on the main
+ * connection
+ */
+
+ return NGX_OK;
+ }
+
+#endif
+
n = ngx_file_aio_read(file->file, buf, 1, file->file_pos, NULL);

if (n == NGX_AGAIN) {
@@ -270,6 +286,23 @@ ngx_http_copy_thread_handler(ngx_thread_

r = file->thread_ctx;

+#if (NGX_HTTP_V2)
+
+ if (r->aio
+ && r->stream
+ && r->stream->connection->connection->sendfile_task == task)
+ {
+ /*
+ * with HTTP/2, another thread operation might be already running
+ * if sendfile() is called as a result of a write event on the main
+ * connection
+ */
+
+ return NGX_OK;
+ }
+
+#endif
+
clcf = ngx_http_get_module_loc_conf(r, ngx_http_core_module);
tp = clcf->thread_pool;

diff --git a/src/http/ngx_http_upstream.c b/src/http/ngx_http_upstream.c
--- a/src/http/ngx_http_upstream.c
+++ b/src/http/ngx_http_upstream.c
@@ -3854,6 +3854,23 @@ ngx_http_upstream_thread_handler(ngx_thr
r = file->thread_ctx;
p = r->upstream->pipe;

+#if (NGX_HTTP_V2)
+
+ if (r->aio
+ && r->stream
+ && r->stream->connection->connection->sendfile_task == task)
+ {
+ /*
+ * with HTTP/2, another thread operation might be already running
+ * if sendfile() is called as a result of a write event on the main
+ * connection
+ */
+
+ return NGX_OK;
+ }
+
+#endif
+
clcf = ngx_http_get_module_loc_conf(r, ngx_http_core_module);
tp = clcf->thread_pool;


[...]

No changes in the second patch.

--
Maxim Dounin
http://mdounin.ru/
_______________________________________________
nginx-devel mailing list
nginx-devel@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx-devel