Welcome! Log In Create A New Profile

Advanced

Re: Memory Management ( > 25GB memory usage)

June 03, 2013 10:13AM
Thanks Maxim for you answer!

Maxim Dounin Wrote:
-------------------------------------------------------
> Hello!
>
> On Mon, Jun 03, 2013 at 08:57:21AM -0400, Belly wrote:
>
> > Hello nginx!
> >
> > I have one worker-process, which uses over 25GB memory (and doesn't
> stop to
> > do that).
> > My configuration is... let's say special:
> >
> > So there is nginx, which proxies all requests to the PHP backend and
> the PHP
> > backend sends a large request back to nginx. I set the
> fastcgi_buffers very
> > enormous huge to avoid nginx creating temporary files on my disk -
> which
> > would result in high CPU load.
> >
> > Here is my configuration: (reduced to the problem)
> >
> > worker_processes 1;
> > worker_rlimit_nofile 80000;
> > worker_priority -20;
> >
> > events {
> > worker_connections 10240;
> > multi_accept on;
> > }
> > # ...
> > # fastcgi settings
> > fastcgi_buffers 20480 1k;
>
> Just a side note: each buffer structure takes about 100 bytes of
> memory on 64-bit platforms, and using 1k buffers results in about
> 10% overhead just because of this.
>

Very interesting! - Didn't know that... Thanks!

> > fastcgi_connect_timeout 30;
> > fastcgi_read_timeout 30;
> > fastcgi_send_timeout 30;
> > fastcgi_keep_conn on;
> > upstream php-backend {
> > server 127.0.0.1:9000;
> > keepalive 10000;
> > }
> >
> >
> > As you can see the buffers are extreme large, to avoid disk
> buffering. The
> > problem is that nginx doesn't free the buffers. It just eats and
> eats. I
> > know it's my fault and not nginx' fault. What am I doing wrong?
> >
> > The response of my php backend could be from 1k to 300mb.
>
> With your settings each connection can allocate up to 20M of
> buffers. That is, 1500k connections are enough to allocate 25G of
> memory. So the basic question is - how many connections are open?
>

1000 - 2000... got your point.

> With pessimistic assumption of 10k connections as per
> worker_connections, you configuration will result in more than
> 200G memory used.
>
> > What is the best setting for my situation?
>
> I would recommend using "fastcgi_max_temp_file_size 0;" if you
> want to disable disk buffering (see [1]), and configuring some
> reasonable number of reasonably sized fastcgi_buffers. I would
> recommend starting tuning with something like 32 x 64k buffers.
>
> [1] http://nginx.org/r/fastcgi_max_temp_file_size
>

I read about fastcgi_max_temp_file_size, but I'm a bit afraid of.
fastcgi_max_temp_file_size 0; states that data will be transfered synchronously. What does it mean exactly? Is it faster/better than disk buffering? Nginx is built in an asynchronous way. What happens if a worker will do a synchronous job inside an asynchronous one? Will it block the event loop?

> --
> Maxim Dounin
> http://nginx.org/en/donation.html
>
> _______________________________________________
> nginx mailing list
> nginx@nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx
SubjectAuthorPosted

Memory Management ( > 25GB memory usage)

BellyJune 03, 2013 08:57AM

Re: Memory Management ( > 25GB memory usage)

shahzaib1232June 03, 2013 09:06AM

Re: Memory Management ( > 25GB memory usage)

BellyJune 03, 2013 09:13AM

Re: Memory Management ( > 25GB memory usage)

Maxim DouninJune 03, 2013 09:52AM

Re: Memory Management ( > 25GB memory usage)

BellyJune 03, 2013 10:13AM

Re: Memory Management ( > 25GB memory usage)

Maxim DouninJune 03, 2013 11:14AM

Re: Memory Management ( > 25GB memory usage)

Jonathan VanascoJune 03, 2013 11:18AM

Re: Memory Management ( > 25GB memory usage)

BellyJune 04, 2013 03:48AM



Sorry, only registered users may post in this forum.

Click here to login

Online Users

Guests: 102
Record Number of Users: 7 on March 06, 2014
Record Number of Guests: 229 on August 01, 2014
Powered by nginx    Powered by FreeBSD    PHP Powered    Powered by Percona     ipv6 ready