Welcome! Log In Create A New Profile

Advanced

Re: Is it possible that nginx will not buffer the client body?

All files from this thread

File Name File Size   Posted by Date  
no_buffer.patch 38.8 KB open | download 姚伟斌 01/13/2013 Read message
100 bytes open | download Weibin Yao 02/26/2013 Read message
no_buffer_v5.patch 39.2 KB open | download Weibin Yao 03/14/2013 Read message
Pasi Kärkkäinen
February 22, 2013 05:52AM
On Fri, Feb 22, 2013 at 11:25:24AM +0200, Pasi Kärkkäinen wrote:
> On Fri, Feb 22, 2013 at 10:06:11AM +0800, Weibin Yao wrote:
> > Use the patch I attached in this mail thread instead, don't use the pull
> > request patch which is for tengine.Â
> > Thanks.
> >
>
> Oh sorry I missed that attachment. It seems to apply and build OK.
> I'll start testing it.
>

I added the patch on top of nginx 1.2.7 and enabled the following options:

client_body_postpone_sending 64k;
proxy_request_buffering off;

after that connections through the nginx reverse proxy started failing
with errors like this:

[error] 29087#0: *49 upstream prematurely closed connection while reading response header from upstream
[error] 29087#0: *60 upstream sent invalid header while reading response header from upstream

And the services are unusable.

Commenting out the two config options above makes nginx happy again.
Any idea what causes that? Any tips how to troubleshoot it?

Thanks!

-- Pasi


>
> > 2013/2/22 Pasi KÀrkkÀinen <[1]pasik@iki.fi>
> >
> > On Fri, Jan 18, 2013 at 10:38:21AM +0200, Pasi KÀrkkÀinen wrote:
> > > On Thu, Jan 17, 2013 at 11:15:58AM +0800, ?????? wrote:
> > > > Â Â Yes. It should work for any request method.
> > > >
> > >
> > > Great, thanks, I'll let you know how it works for me. Probably in two
> > weeks or so.
> > >
> >
> > Hi,
> >
> > Adding the tengine pull request 91 on top of nginx 1.2.7 doesn't work:
> >
> > cc1: warnings being treated as errors
> > src/http/ngx_http_request_body.c: In function
> > 'ngx_http_read_non_buffered_client_request_body':
> > src/http/ngx_http_request_body.c:506: error: implicit declaration of
> > function 'ngx_http_top_input_body_filter'
> > make[1]: *** [objs/src/http/ngx_http_request_body.o] Error 1
> > make[1]: Leaving directory `/root/src/nginx/nginx-1.2.7'
> > make: *** [build] Error 2
> >
> > ngx_http_top_input_body_filter() cannot be found from any .c/.h files..
> > Which other patches should I apply?
> >
> > Perhaps this?
> > [2]https://github.com/cfsego/limit_upload_rate/blob/master/for-nginx.patch
> >
> > Thanks,
> > -- Pasi
> >
> > >
> > > > Â Â 2013/1/16 Pasi KÃ*â*¬rkkÃ*â*¬inen <[1][3]pasik@iki.fi>
> > > >
> > > > Â Â Â On Sun, Jan 13, 2013 at 08:22:17PM +0800, ?????? wrote:
> > > > Â Â Â > Ã* Â Ã* This patch should work between nginx-1.2.6 and
> > nginx-1.3.8.
> > > > Â Â Â > Ã* Â Ã* The documentation is here:
> > > >
> > > > Â Â Â > Ã* Â Ã* ## client_body_postpone_sending ##
> > > > Â Â Â > Ã* Â Ã* Syntax: **client_body_postpone_sending** `size`
> > > > Â Â Â > Ã* Â Ã* Default: 64k
> > > > Â Â Â > Ã* Â Ã* Context: `http, server, location`
> > > > Â Â Â > Ã* Â Ã* If you specify the `proxy_request_buffering` or
> > > > Â Â Â > Ã* Â Ã* `fastcgi_request_buffering` to be off, Nginx will
> > send the body
> > > > Â Â Â to backend
> > > > Â Â Â > Ã* Â Ã* when it receives more than `size` data or the
> > whole request body
> > > > Â Â Â has been
> > > > Â Â Â > Ã* Â Ã* received. It could save the connection and reduce
> > the IO number
> > > > Â Â Â with
> > > > Â Â Â > Ã* Â Ã* backend.
> > > > Â Â Â >
> > > > Â Â Â > Ã* Â Ã* ## proxy_request_buffering ##
> > > > Â Â Â > Ã* Â Ã* Syntax: **proxy_request_buffering** `on | off`
> > > > Â Â Â > Ã* Â Ã* Default: `on`
> > > > Â Â Â > Ã* Â Ã* Context: `http, server, location`
> > > > Â Â Â > Ã* Â Ã* Specify the request body will be buffered to the
> > disk or not. If
> > > > Â Â Â it's off,
> > > > Â Â Â > Ã* Â Ã* the request body will be stored in memory and sent
> > to backend
> > > > Â Â Â after Nginx
> > > > Â Â Â > Ã* Â Ã* receives more than `client_body_postpone_sending`
> > data. It could
> > > > Â Â Â save the
> > > > Â Â Â > Ã* Â Ã* disk IO with large request body.
> > > > Â Â Â >
> > > > Â Â Â >
> > > > Â Â Â > Ã* Â Ã* Â Ã* Â Ã* Â Ã* Â Ã* Â Note that, if you specify it
> > to be off, the nginx
> > > > Â Â Â retry mechanism
> > > > Â Â Â > Ã* Â Ã* with unsuccessful response will be broken after
> > you sent part of
> > > > Â Â Â the
> > > > Â Â Â > Ã* Â Ã* request to backend. It will just return 500 when
> > it encounters
> > > > Â Â Â such
> > > > Â Â Â > Ã* Â Ã* unsuccessful response. This directive also breaks
> > these
> > > > Â Â Â variables:
> > > > Â Â Â > Ã* Â Ã* $request_body, $request_body_file. You should not
> > use these
> > > > Â Â Â variables any
> > > > Â Â Â > Ã* Â Ã* more while their values are undefined.
> > > > Â Â Â >
> > > >
> > > > Â Â Â Hello,
> > > >
> > > > Â Â Â This patch sounds exactly like what I need aswell!
> > > > Â Â Â I assume it works for both POST and PUT requests?
> > > >
> > > > Â Â Â Thanks,
> > > >
> > > > Â Â Â -- Pasi
> > > >
> > > > Â Â Â > Ã* Â Ã* Â Ã* Hello!
> > > > Â Â Â > Ã* Â Ã* Â Ã* @yaoweibin
> > > > Â Â Â >
> > > > Â Â Â > Ã* Â Ã* Â Ã* Â Ã* If you are eager for this feature, you
> > could try my
> > > > Â Â Â > Ã* Â Ã* Â Ã* Â Ã* patch:
> > [2][2][4]https://github.com/taobao/tengine/pull/91.
> > > > Â Â Â This patch has
> > > > Â Â Â > Ã* Â Ã* Â Ã* Â Ã* been running in our production servers.
> > > > Â Â Â >
> > > > Â Â Â > Ã* Â Ã* Â Ã* what's the nginx version your patch based on?
> > > > Â Â Â > Ã* Â Ã* Â Ã* Thanks!
> > > > Â Â Â > Ã* Â Ã* Â Ã* On Fri, Jan 11, 2013 at 5:17 PM, Ã**Ã*
> > Ã**Ã*°Ã*±Ã*³
> > > > Â Â Â <[3][3][5]yaoweibin@gmail.com> wrote:
> > > > Â Â Â >
> > > > Â Â Â > Ã* Â Ã* Â Ã* Â Ã* I know nginx team are working on it. You
> > can wait for it.
> > > > Â Â Â > Ã* Â Ã* Â Ã* Â Ã* If you are eager for this feature, you
> > could try my
> > > > Â Â Â > Ã* Â Ã* Â Ã* Â Ã* patch:
> > [4][4][6]https://github.com/taobao/tengine/pull/91.
> > > > Â Â Â This patch has
> > > > Â Â Â > Ã* Â Ã* Â Ã* Â Ã* been running in our production servers.
> > > > Â Â Â >
> > > > Â Â Â > Ã* Â Ã* Â Ã* Â Ã* 2013/1/11 li zJay
> > <[5][5][7]zjay1987@gmail.com>
> > > > Â Â Â >
> > > > Â Â Â > Ã* Â Ã* Â Ã* Â Ã* Â Ã* Hello!
> > > > Â Â Â > Ã* Â Ã* Â Ã* Â Ã* Â Ã* is it possible that nginx will not
> > buffer the client
> > > > Â Â Â body before
> > > > Â Â Â > Ã* Â Ã* Â Ã* Â Ã* Â Ã* handle the request to upstream?
> > > > Â Â Â > Ã* Â Ã* Â Ã* Â Ã* Â Ã* we want to use nginx as a reverse
> > proxy to upload very
> > > > Â Â Â very big file
> > > > Â Â Â > Ã* Â Ã* Â Ã* Â Ã* Â Ã* to the upstream, but the default
> > behavior of nginx is to
> > > > Â Â Â save the
> > > > Â Â Â > Ã* Â Ã* Â Ã* Â Ã* Â Ã* whole request to the local disk
> > first before handle it
> > > > Â Â Â to the
> > > > Â Â Â > Ã* Â Ã* Â Ã* Â Ã* Â Ã* upstream, which make the upstream
> > impossible to process
> > > > Â Â Â the file on
> > > > Â Â Â > Ã* Â Ã* Â Ã* Â Ã* Â Ã* the fly when the file is uploading,
> > results in much high
> > > > Â Â Â request
> > > > Â Â Â > Ã* Â Ã* Â Ã* Â Ã* Â Ã* latency and server-side resource
> > consumption.
> > > > Â Â Â > Ã* Â Ã* Â Ã* Â Ã* Â Ã* Thanks!
> > > > Â Â Â > Ã* Â Ã* Â Ã* Â Ã* Â Ã*
> > _______________________________________________
> > > > Â Â Â > Ã* Â Ã* Â Ã* Â Ã* Â Ã* nginx mailing list
> > > > Â Â Â > Ã* Â Ã* Â Ã* Â Ã* Â Ã* [6][6][8]nginx@nginx.org
> > > > Â Â Â > Ã* Â Ã* Â Ã* Â Ã* Â Ã*
> > [7][7][9]http://mailman.nginx.org/mailman/listinfo/nginx
> > > > Â Â Â >
> > > > Â Â Â > Ã* Â Ã* Â Ã* Â Ã* --
> > > > Â Â Â > Ã* Â Ã* Â Ã* Â Ã* Weibin Yao
> > > > Â Â Â > Ã* Â Ã* Â Ã* Â Ã* Developer @ Server Platform Team of
> > Taobao
> > > > Â Â Â > Ã* Â Ã* Â Ã* Â Ã*
> > _______________________________________________
> > > > Â Â Â > Ã* Â Ã* Â Ã* Â Ã* nginx mailing list
> > > > Â Â Â > Ã* Â Ã* Â Ã* Â Ã* [8][8][10]nginx@nginx.org
> > > > Â Â Â > Ã* Â Ã* Â Ã* Â Ã*
> > [9][9][11]http://mailman.nginx.org/mailman/listinfo/nginx
> > > > Â Â Â >
> > > > Â Â Â > Ã* Â Ã* Â Ã*
> > _______________________________________________
> > > > Â Â Â > Ã* Â Ã* Â Ã* nginx mailing list
> > > > Â Â Â > Ã* Â Ã* Â Ã* [10][10][12]nginx@nginx.org
> > > > Â Â Â > Ã* Â Ã* Â Ã*
> > [11][11][13]http://mailman.nginx.org/mailman/listinfo/nginx
> > > > Â Â Â >
> > > > Â Â Â > Ã* Â Ã* --
> > > > Â Â Â > Ã* Â Ã* Weibin Yao
> > > > Â Â Â > Ã* Â Ã* Developer @ Server Platform Team of Taobao
> > > > Â Â Â >
> > > > Â Â Â > References
> > > > Â Â Â >
> > > > Â Â Â > Ã* Â Ã* Visible links
> > > > Â Â Â > Ã* Â Ã* 1. mailto:[12][14]zjay1987@gmail.com
> > > > Â Â Â > Ã* Â Ã* 2.
> > [13][15]https://github.com/taobao/tengine/pull/91
> > > > Â Â Â > Ã* Â Ã* 3. mailto:[14][16]yaoweibin@gmail.com
> > > > Â Â Â > Ã* Â Ã* 4.
> > [15][17]https://github.com/taobao/tengine/pull/91
> > > > Â Â Â > Ã* Â Ã* 5. mailto:[16][18]zjay1987@gmail.com
> > > > Â Â Â > Ã* Â Ã* 6. mailto:[17][19]nginx@nginx.org
> > > > Â Â Â > Ã* Â Ã* 7.
> > [18][20]http://mailman.nginx.org/mailman/listinfo/nginx
> > > > Â Â Â > Ã* Â Ã* 8. mailto:[19][21]nginx@nginx.org
> > > > Â Â Â > Ã* Â Ã* 9.
> > [20][22]http://mailman.nginx.org/mailman/listinfo/nginx
> > > > Â Â Â > Ã* Â 10. mailto:[21][23]nginx@nginx.org
> > > > Â Â Â > Ã* Â 11.
> > [22][24]http://mailman.nginx.org/mailman/listinfo/nginx
> > > >
> > > > Â Â Â > _______________________________________________
> > > > Â Â Â > nginx mailing list
> > > > Â Â Â > [23][25]nginx@nginx.org
> > > > Â Â Â > [24][26]http://mailman.nginx.org/mailman/listinfo/nginx
> > > >
> > > > Â Â Â _______________________________________________
> > > > Â Â Â nginx mailing list
> > > > Â Â Â [25][27]nginx@nginx.org
> > > > Â Â Â [26][28]http://mailman.nginx.org/mailman/listinfo/nginx
> > > >
> > > > Â Â --
> > > > Â Â Weibin Yao
> > > > Â Â Developer @ Server Platform Team of Taobao
> > > >
> > > > References
> > > >
> > > > Â Â Visible links
> > > > Â Â 1. mailto:[29]pasik@iki.fi
> > > > Â Â 2. [30]https://github.com/taobao/tengine/pull/91
> > > > Â Â 3. mailto:[31]yaoweibin@gmail.com
> > > > Â Â 4. [32]https://github.com/taobao/tengine/pull/91
> > > > Â Â 5. mailto:[33]zjay1987@gmail.com
> > > > Â Â 6. mailto:[34]nginx@nginx.org
> > > > Â Â 7. [35]http://mailman.nginx.org/mailman/listinfo/nginx
> > > > Â Â 8. mailto:[36]nginx@nginx.org
> > > > Â Â 9. [37]http://mailman.nginx.org/mailman/listinfo/nginx
> > > > Â 10. mailto:[38]nginx@nginx.org
> > > > Â 11. [39]http://mailman.nginx.org/mailman/listinfo/nginx
> > > > Â 12. mailto:[40]zjay1987@gmail.com
> > > > Â 13. [41]https://github.com/taobao/tengine/pull/91
> > > > Â 14. mailto:[42]yaoweibin@gmail.com
> > > > Â 15. [43]https://github.com/taobao/tengine/pull/91
> > > > Â 16. mailto:[44]zjay1987@gmail.com
> > > > Â 17. mailto:[45]nginx@nginx.org
> > > > Â 18. [46]http://mailman.nginx.org/mailman/listinfo/nginx
> > > > Â 19. mailto:[47]nginx@nginx.org
> > > > Â 20. [48]http://mailman.nginx.org/mailman/listinfo/nginx
> > > > Â 21. mailto:[49]nginx@nginx.org
> > > > Â 22. [50]http://mailman.nginx.org/mailman/listinfo/nginx
> > > > Â 23. mailto:[51]nginx@nginx.org
> > > > Â 24. [52]http://mailman.nginx.org/mailman/listinfo/nginx
> > > > Â 25. mailto:[53]nginx@nginx.org
> > > > Â 26. [54]http://mailman.nginx.org/mailman/listinfo/nginx
> > >
> > > > _______________________________________________
> > > > nginx mailing list
> > > > [55]nginx@nginx.org
> > > > [56]http://mailman.nginx.org/mailman/listinfo/nginx
> > >
> > > _______________________________________________
> > > nginx mailing list
> > > [57]nginx@nginx.org
> > > [58]http://mailman.nginx.org/mailman/listinfo/nginx
> >
> > _______________________________________________
> > nginx mailing list
> > [59]nginx@nginx.org
> > [60]http://mailman.nginx.org/mailman/listinfo/nginx
> >
> > --
> > Weibin Yao
> > Developer @ Server Platform Team of Taobao
> >
> > References
> >
> > Visible links
> > 1. mailto:pasik@iki.fi
> > 2. https://github.com/cfsego/limit_upload_rate/blob/master/for-nginx.patch
> > 3. mailto:pasik@iki.fi
> > 4. https://github.com/taobao/tengine/pull/91
> > 5. mailto:yaoweibin@gmail.com
> > 6. https://github.com/taobao/tengine/pull/91
> > 7. mailto:zjay1987@gmail.com
> > 8. mailto:nginx@nginx.org
> > 9. http://mailman.nginx.org/mailman/listinfo/nginx
> > 10. mailto:nginx@nginx.org
> > 11. http://mailman.nginx.org/mailman/listinfo/nginx
> > 12. mailto:nginx@nginx.org
> > 13. http://mailman.nginx.org/mailman/listinfo/nginx
> > 14. mailto:zjay1987@gmail.com
> > 15. https://github.com/taobao/tengine/pull/91
> > 16. mailto:yaoweibin@gmail.com
> > 17. https://github.com/taobao/tengine/pull/91
> > 18. mailto:zjay1987@gmail.com
> > 19. mailto:nginx@nginx.org
> > 20. http://mailman.nginx.org/mailman/listinfo/nginx
> > 21. mailto:nginx@nginx.org
> > 22. http://mailman.nginx.org/mailman/listinfo/nginx
> > 23. mailto:nginx@nginx.org
> > 24. http://mailman.nginx.org/mailman/listinfo/nginx
> > 25. mailto:nginx@nginx.org
> > 26. http://mailman.nginx.org/mailman/listinfo/nginx
> > 27. mailto:nginx@nginx.org
> > 28. http://mailman.nginx.org/mailman/listinfo/nginx
> > 29. mailto:pasik@iki.fi
> > 30. https://github.com/taobao/tengine/pull/91
> > 31. mailto:yaoweibin@gmail.com
> > 32. https://github.com/taobao/tengine/pull/91
> > 33. mailto:zjay1987@gmail.com
> > 34. mailto:nginx@nginx.org
> > 35. http://mailman.nginx.org/mailman/listinfo/nginx
> > 36. mailto:nginx@nginx.org
> > 37. http://mailman.nginx.org/mailman/listinfo/nginx
> > 38. mailto:nginx@nginx.org
> > 39. http://mailman.nginx.org/mailman/listinfo/nginx
> > 40. mailto:zjay1987@gmail.com
> > 41. https://github.com/taobao/tengine/pull/91
> > 42. mailto:yaoweibin@gmail.com
> > 43. https://github.com/taobao/tengine/pull/91
> > 44. mailto:zjay1987@gmail.com
> > 45. mailto:nginx@nginx.org
> > 46. http://mailman.nginx.org/mailman/listinfo/nginx
> > 47. mailto:nginx@nginx.org
> > 48. http://mailman.nginx.org/mailman/listinfo/nginx
> > 49. mailto:nginx@nginx.org
> > 50. http://mailman.nginx.org/mailman/listinfo/nginx
> > 51. mailto:nginx@nginx.org
> > 52. http://mailman.nginx.org/mailman/listinfo/nginx
> > 53. mailto:nginx@nginx.org
> > 54. http://mailman.nginx.org/mailman/listinfo/nginx
> > 55. mailto:nginx@nginx.org
> > 56. http://mailman.nginx.org/mailman/listinfo/nginx
> > 57. mailto:nginx@nginx.org
> > 58. http://mailman.nginx.org/mailman/listinfo/nginx
> > 59. mailto:nginx@nginx.org
> > 60. http://mailman.nginx.org/mailman/listinfo/nginx
>
> > _______________________________________________
> > nginx mailing list
> > nginx@nginx.org
> > http://mailman.nginx.org/mailman/listinfo/nginx
>
> _______________________________________________
> nginx mailing list
> nginx@nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Subject Author Posted

Is it possible that nginx will not buffer the client body?

lm011111 January 11, 2013 02:22AM

Re: Is it possible that nginx will not buffer the client body?

姚伟斌 January 11, 2013 04:18AM

Re: Is it possible that nginx will not buffer the client body?

lm011111 January 13, 2013 04:02AM

Re: Is it possible that nginx will not buffer the client body? Attachments

姚伟斌 January 13, 2013 07:24AM

Re: Is it possible that nginx will not buffer the client body?

Pasi Kärkkäinen January 16, 2013 10:16AM

Re: Is it possible that nginx will not buffer the client body?

姚伟斌 January 16, 2013 10:18PM

Re: Is it possible that nginx will not buffer the client body?

Pasi Kärkkäinen January 18, 2013 03:40AM

Re: Is it possible that nginx will not buffer the client body?

Pasi Kärkkäinen February 21, 2013 03:10PM

Re: Is it possible that nginx will not buffer the client body?

Weibin Yao February 21, 2013 09:08PM

Re: Is it possible that nginx will not buffer the client body?

Pasi Kärkkäinen February 22, 2013 04:26AM

Re: Is it possible that nginx will not buffer the client body?

Pasi Kärkkäinen February 22, 2013 05:52AM

Re: Is it possible that nginx will not buffer the client body?

Weibin Yao February 24, 2013 09:14PM

Re: Is it possible that nginx will not buffer the client body?

Pasi Kärkkäinen February 25, 2013 05:14AM

Re: Is it possible that nginx will not buffer the client body? Attachments

Weibin Yao February 26, 2013 09:14AM

Re: Is it possible that nginx will not buffer the client body?

Pasi Kärkkäinen March 05, 2013 08:18AM

Re: Is it possible that nginx will not buffer the client body?

double March 07, 2013 12:25PM

Re: Is it possible that nginx will not buffer the client body?

Pasi Kärkkäinen March 07, 2013 12:50PM

Re: Is it possible that nginx will not buffer the client body?

Pasi Kärkkäinen March 14, 2013 04:46AM

Re: Is it possible that nginx will not buffer the client body?

Pasi Kärkkäinen March 08, 2013 08:38AM

Re: Is it possible that nginx will not buffer the client body? Attachments

Weibin Yao March 14, 2013 01:18AM

Re: Is it possible that nginx will not buffer the client body?

Pasi Kärkkäinen March 14, 2013 04:40AM

Re: Is it possible that nginx will not buffer the client body?

Pasi Kärkkäinen March 20, 2013 01:04PM

Re: Is it possible that nginx will not buffer the client body?

double February 28, 2013 10:16AM

Re: Is it possible that nginx will not buffer the client body?

EDevil February 28, 2013 12:38PM

Re: Is it possible that nginx will not buffer the client body?

Maxim Dounin February 28, 2013 01:14PM

Re: Is it possible that nginx will not buffer the client body?

double March 05, 2013 06:24AM

Re: Is it possible that nginx will not buffer the client body?

Pasi Kärkkäinen March 14, 2013 10:14AM

Re: Is it possible that nginx will not buffer the client body?

Weibin Yao March 18, 2013 12:22AM

Re: Is it possible that nginx will not buffer the client body?

double April 04, 2013 04:30AM

Re: Is it possible that nginx will not buffer the client body?

Valentin V. Bartenev February 28, 2013 01:52PM

Re: Is it possible that nginx will not buffer the client body?

EDevil February 28, 2013 02:04PM

Re: Is it possible that nginx will not buffer the client body?

Pasi Kärkkäinen March 01, 2013 08:24AM

Re: Is it possible that nginx will not buffer the client body?

double March 01, 2013 10:17AM

Re: Is it possible that nginx will not buffer the client body?

daveyfx January 11, 2013 02:31PM

Re: Is it possible that nginx will not buffer the client body?

double February 26, 2013 04:10PM

Re: Is it possible that nginx will not buffer the client body?

double April 18, 2013 05:27AM



Sorry, only registered users may post in this forum.

Click here to login

Online Users

Guests: 54
Record Number of Users: 6 on February 13, 2018
Record Number of Guests: 421 on December 02, 2018
Powered by nginx      Powered by FreeBSD      PHP Powered      Powered by MariaDB      ipv6 ready