Yes, it was timeouts, but that could have been fixed in the newer nginx
versions. I'm just wondering why
the ram usage hasn't occured before and and is now.. It never used to
use much at all and now it's using an
exorbitant amount of ram with nothing major changed.. That's why I
figured something was wrong.
It takes it about 12 hours or so to use 6-7gb of ram. That is another
reason why I thought something might be wrong.
I'll try changing the buffers and watching the RAM usage again and see
what happens.
Thank you, and Igor, for your inputs.
Paul
Cliff Wells wrote:
> On Mon, 2009-08-31 at 16:52 -0400, Paul wrote:
>
>> Had some issues with people uploading/downloading files before on 0.6
>> and the buffers fixed it..
>>
>
> What sort of issues? Timeouts or "client body too large"? I regularly
> upload 10MB+ CSV files to a proxied application with the following:
>
> client_max_body_size 20m;
> client_body_buffer_size 128k;
> proxy_connect_timeout 90;
> proxy_send_timeout 90;
> proxy_read_timeout 90;
> proxy_buffer_size 4k;
> proxy_buffers 4 32k;
> proxy_busy_buffers_size 64k;
> proxy_temp_file_write_size 64k;
>
>
> Cliff
>
>
>> What would you suggest I change the config to? I will try and let you
>> know if I still see the problem.
>> Thank you.
>>
>>
>> Igor Sysoev wrote:
>>
>>> On Mon, Aug 31, 2009 at 04:17:03PM -0400, Paul wrote:
>>>
>>>
>>>
>>>> I know, but this problem has never occured until recently.. Once the
>>>> request is done, it should remove the memory allocation, but it looks
>>>> like maybe it isn't?
>>>>
>>>>
>>> No. These workers run since Aug 7 and handle up to 3,800r/s:
>>>
>>> ps ax -o pid,ppid,%cpu,vsz,wchan,start,command|egrep '(nginx|PID)'
>>> PID PPID %CPU VSZ WCHAN STARTED COMMAND
>>> 42412 51429 16.7 372452 kqread 7Aug09 nginx: worker process (nginx)
>>> 42413 51429 18.2 372452 kqread 7Aug09 nginx: worker process (nginx)
>>> 42414 51429 0.0 291556 kqread 7Aug09 nginx: cache manager process (nginx)
>>> 51429 1 0.0 291556 pause 28Jul09 nginx: master process /usr/local/nginx/ng
>>>
>>>
>>>
>>>> The only difference in a month ago and now is that we have more server
>>>> entries, and more requests per second. It used to not even use a gig of
>>>> ram doing 200 requests/sec and now it keeps using more and more ram
>>>> until it fills the entire ram and swap and errors out 8gb+ ..
>>>> What would you suggest?
>>>>
>>>>
>>> Just 250 simultaneous proxied connections take 8G. And 250 connections
>>> are too little in modern world. Why at all you have set up so huge buffers ?
>>>
>>>
>>>
>>>> Igor Sysoev wrote:
>>>>
>>>>
>>>>> On Sun, Aug 30, 2009 at 09:45:55PM -0400, Paul wrote:
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>> I had nginx 0.6.32 and just upgraded to 0.7.61 and still have the same
>>>>>> memleak issue.
>>>>>> What happens is as soon as I start nginx, it starts using ram and
>>>>>> continues to use more and more and more over a 16 hour period is
>>>>>> consumes 8gb ram and all the swap and then errors out because it cant
>>>>>> use any more.
>>>>>> This never used to happen until recently and the only difference at all
>>>>>> in the config is more server entries.
>>>>>>
>>>>>> here's the config:
>>>>>>
>>>>>> user www www;
>>>>>>
>>>>>> worker_processes 16;
>>>>>> error_log logs/error.log;
>>>>>> worker_rlimit_nofile 65000;
>>>>>>
>>>>>> events
>>>>>> {
>>>>>>
>>>>>> worker_connections 40000;
>>>>>> }
>>>>>>
>>>>>> ####### HTTP SETTING
>>>>>> http
>>>>>> {
>>>>>> access_log off;
>>>>>> log_format alot '$remote_addr - $remote_user [$time_local] '
>>>>>> '"$request" $status $body_bytes_sent '
>>>>>> '"$http_referer" "$http_user_agent" "$http_accept"
>>>>>> $connection';
>>>>>> sendfile on;
>>>>>> tcp_nopush on;
>>>>>> tcp_nodelay on;
>>>>>> keepalive_timeout 0;
>>>>>> output_buffers 16 128k;
>>>>>> server_tokens off;
>>>>>> ssl_verify_client off;
>>>>>> ssl_session_timeout 10m;
>>>>>> # ssl_session_cache shared:SSL:500000;
>>>>>> include /usr/local/nginx/conf/mime.types;
>>>>>> default_type application/octet-stream;
>>>>>>
>>>>>> # cache_max_size 24;
>>>>>>
>>>>>> gzip on;
>>>>>> gzip_min_length 512;
>>>>>> gzip_buffers 64 32k;
>>>>>> gzip_types text/plain text/html text/xhtml text/css text/js;
>>>>>>
>>>>>> proxy_buffering on;
>>>>>> proxy_buffer_size 32m;
>>>>>> proxy_buffers 16 32m;
>>>>>>
>>>>>>
>>>>>>
>>>>> These settings allocate 32M buffer for each proxied request.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>> proxy_busy_buffers_size 64m;
>>>>>> proxy_temp_file_write_size 2048m;
>>>>>> proxy_intercept_errors on;
>>>>>> proxy_ssl_session_reuse off;
>>>>>> proxy_read_timeout 120;
>>>>>> proxy_connect_timeout 60;
>>>>>> proxy_send_timeout 120;
>>>>>> client_body_buffer_size 32m;
>>>>>>
>>>>>>
>>>>>>
>>>>> This setting allocates 32M buffer for each request with body.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>> client_header_buffer_size 64k;
>>>>>> large_client_header_buffers 16 64k;
>>>>>> client_max_body_size 16m;
>>>>>>
>>>>>>
>>>>>> server
>>>>>> {
>>>>>> listen 1.2.3.4:80;
>>>>>> location /
>>>>>> {
>>>>>>
>>>>>> proxy_pass http://3.4.5.6;
>>>>>> proxy_redirect http://3.4.5.6/
>>>>>> http://$http_host/;
>>>>>> proxy_redirect default;
>>>>>> proxy_set_header Host
>>>>>> $host; ##Forwards host along
>>>>>> proxy_set_header X-Real-IP
>>>>>> $remote_addr; ##Sends realip to customer svr
>>>>>> proxy_set_header X-Forwarded-For
>>>>>> $remote_addr; ##Sends realip to customer svr
>>>>>> }
>>>>>> }
>>>>>> server
>>>>>> {
>>>>>> listen 1.2.3.4:443;
>>>>>>
>>>>>> ssl on;
>>>>>> ssl_certificate
>>>>>> /usr/local/nginx/conf/whatever.com.crt;
>>>>>> ssl_certificate_key
>>>>>> /usr/local/nginx/conf/whatever.com.key;
>>>>>> ssl_protocols SSLv3;
>>>>>> ssl_ciphers HIGH:!ADH;
>>>>>> ssl_prefer_server_ciphers on;
>>>>>> location /
>>>>>> {
>>>>>> proxy_pass https://3.4.5.6;
>>>>>> proxy_redirect https://3.4.5.6/
>>>>>> http://$http_host/;
>>>>>> proxy_redirect default;
>>>>>> proxy_set_header Host
>>>>>> $host; ##Forwards host along
>>>>>> proxy_set_header X-Real-IP
>>>>>> $remote_addr; ##Sends realip to customer svr
>>>>>> proxy_set_header X-Forwarded-For
>>>>>> $remote_addr; ##Sends realip to customer svr
>>>>>> proxy_set_header X-FORWARDED_PROTO https;
>>>>>> }
>>>>>> }
>>>>>>
>>>>>> And these server entries repeated about 60 or so times and that's it.
>>>>>> When it was around 40 we never had a memleak issue.
>>>>>>
>>>>>> This is on Linux, kernel is 2.6.25
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>>
>>>>>> --
>>>>>> GloboTech Communications
>>>>>> Phone: 1-514-907-0050 x 215
>>>>>> Toll Free: 1-(888)-GTCOMM1
>>>>>> Fax: 1-(514)-907-0750
>>>>>> paul@gtcomm.net
>>>>>> http://www.gtcomm.net
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>> --
>>>> GloboTech Communications
>>>> Phone: 1-514-907-0050 x 215
>>>> Toll Free: 1-(888)-GTCOMM1
>>>> Fax: 1-(514)-907-0750
>>>> paul@gtcomm.net
>>>> http://www.gtcomm.net
>>>>
>>>>
>>>>
>>>
>>>
--
GloboTech Communications
Phone: 1-514-907-0050 x 215
Toll Free: 1-(888)-GTCOMM1
Fax: 1-(514)-907-0750
paul@gtcomm.net
http://www.gtcomm.net
