Welcome! Log In Create A New Profile

Advanced

Re: Memory Management ( > 25GB memory usage)

Maxim Dounin
June 03, 2013 09:52AM
Hello!

On Mon, Jun 03, 2013 at 08:57:21AM -0400, Belly wrote:

> Hello nginx!
>
> I have one worker-process, which uses over 25GB memory (and doesn't stop to
> do that).
> My configuration is... let's say special:
>
> So there is nginx, which proxies all requests to the PHP backend and the PHP
> backend sends a large request back to nginx. I set the fastcgi_buffers very
> enormous huge to avoid nginx creating temporary files on my disk - which
> would result in high CPU load.
>
> Here is my configuration: (reduced to the problem)
>
> worker_processes 1;
> worker_rlimit_nofile 80000;
> worker_priority -20;
>
> events {
> worker_connections 10240;
> multi_accept on;
> }
> # ...
> # fastcgi settings
> fastcgi_buffers 20480 1k;

Just a side note: each buffer structure takes about 100 bytes of
memory on 64-bit platforms, and using 1k buffers results in about
10% overhead just because of this.

> fastcgi_connect_timeout 30;
> fastcgi_read_timeout 30;
> fastcgi_send_timeout 30;
> fastcgi_keep_conn on;
> upstream php-backend {
> server 127.0.0.1:9000;
> keepalive 10000;
> }
>
>
> As you can see the buffers are extreme large, to avoid disk buffering. The
> problem is that nginx doesn't free the buffers. It just eats and eats. I
> know it's my fault and not nginx' fault. What am I doing wrong?
>
> The response of my php backend could be from 1k to 300mb.

With your settings each connection can allocate up to 20M of
buffers. That is, 1500k connections are enough to allocate 25G of
memory. So the basic question is - how many connections are open?

With pessimistic assumption of 10k connections as per
worker_connections, you configuration will result in more than
200G memory used.

> What is the best setting for my situation?

I would recommend using "fastcgi_max_temp_file_size 0;" if you
want to disable disk buffering (see [1]), and configuring some
reasonable number of reasonably sized fastcgi_buffers. I would
recommend starting tuning with something like 32 x 64k buffers.

[1] http://nginx.org/r/fastcgi_max_temp_file_size

--
Maxim Dounin
http://nginx.org/en/donation.html

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Subject Author Posted

Memory Management ( > 25GB memory usage)

Belly June 03, 2013 08:57AM

Re: Memory Management ( > 25GB memory usage)

shahzaib1232 June 03, 2013 09:06AM

Re: Memory Management ( > 25GB memory usage)

Belly June 03, 2013 09:13AM

Re: Memory Management ( > 25GB memory usage)

Maxim Dounin June 03, 2013 09:52AM

Re: Memory Management ( > 25GB memory usage)

Belly June 03, 2013 10:13AM

Re: Memory Management ( > 25GB memory usage)

Maxim Dounin June 03, 2013 11:14AM

Re: Memory Management ( > 25GB memory usage)

Jonathan Vanasco June 03, 2013 11:18AM

Re: Memory Management ( > 25GB memory usage)

Belly June 04, 2013 03:48AM



Sorry, only registered users may post in this forum.

Click here to login

Online Users

Guests: 91
Record Number of Users: 6 on February 13, 2018
Record Number of Guests: 421 on December 02, 2018
Powered by nginx      Powered by FreeBSD      PHP Powered      Powered by MariaDB      ipv6 ready