On 1 Fev 2013 16h25 CET, mdounin@mdounin.ru wrote:

> Hello!

Hello Maxim,


>> I made some tests and definitely we cannot use in our product, it
>> takes too much time and resources.
>>
>> Tested on an EC m1.medium instance.
>>
>> HOSTS,DATE,COMMAND,CPU_PERCENTAGE,CPU_SYSTEM,CPU_USER,ELAPSED_TIME,IO_PG_FAULTS,ICONTEXT_SWITCHING,VCONTEXT_SWITCHING,MAX_MEMORY
>> 5001,01.Feb.2013 00:18:33,/usr/sbin/nginx -s
>> reload,92%,0.60,1.84,0:02.62,0,3296,1,138528 10001,01.Feb.2013
>> 00:19:32,/usr/sbin/nginx -s
>> reload,93%,1.67,5.80,0:08.00,0,11627,1,406804 20001,01.Feb.2013
>> 00:20:23,/usr/sbin/nginx -s
>> reload,93%,4.17,13.68,0:19.16,0,25221,1,945164 50001,01.Feb.2013
>> 00:22:02,/usr/sbin/nginx -s
>> reload,60%,13.24,36.37,1:22.46,14,68338,87121,2288668
>>
>> As you can see 50k hosts take more than one minute. That would be
>> acceptable if it weren't for the fact that it uses up a lot of CPU
>> and memory.
>
> 50k ssl hosts is a lot, actually, and loading 50k certificates
> in 1 minute on a cheap virtual server looks reasonable for me.
> Especially if you consider total cost of 50k ssl certs.

Yes, but they're the clients/users certs. Not ours.

> Please also make sure you are testing recent enough nginx. In
> paticular, the following commit speeds up ssl loading
> about 2x here (available in nginx 1.3.11):

perusio@dev:~$ /usr/sbin/nginx -V
nginx version: nginx/1.3.11
built by gcc 4.7.2 (Debian 4.7.2-5)
TLS SNI support enabled
configure arguments: --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-client-body-temp-path=/var/lib/nginx/body --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-log-path=/var/log/nginx/access.log --http-proxy-temp-path=/var/lib/nginx/proxy --http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --lock-path=/var/lock/nginx.lock --pid-path=/var/run/nginx.pid --with-debug --without-mail_pop3_module --without-mail_imap_module --without-mail_smtp_module --without-http_fastcgi_module --without-http_autoindex_module --without-http_split_clients_module --without-http_memcached_module --without-http_scgi_module --without-http_browser_module --without-http_split_clients_module --without-http_autoindex_module --without-http_userid_module --without-http_ssi_module --without-select_module --without-poll_module --with-http_gzip_static_module --with-http_realip_module --with-http_stub_status_module --with-http_ssl_module --with-i
pv6 --with-file-aio

> http://trac.nginx.org/nginx/changeset/5004/nginx
>
> The real limiting factor with many server blocks seems to be
> memory used for each server{}. This probably needs some
> optimization.

Looking at htop, the config parsing is taking a lot of time. I know
that because I also do a nginx -t before. It's just that I ommited
that from the mail. Here they are:

HOSTS,DATE,COMMAND,CPU_PERCENTAGE,CPU_SYSTEM,CPU_USER,ELAPSED_TIME,IO_PG_FAULTS,ICONTEXT_SWITCHING,VCONTEXT_SWITCHING,MAX_MEMORY
5001,01.Feb.2013 00:18:31,/usr/sbin/nginx_ensite 03805-08805.test-ssl.local.conf,93%,0.59,1.85,0:02.61,0,3450,17,138532
10001,01.Feb.2013 00:19:24,/usr/sbin/nginx_ensite 08806-18806.test-ssl.local.conf,93%,1.81,5.63,0:07.95,0,10684,16,406804
20001,01.Feb.2013 00:20:04,/usr/sbin/nginx_ensite 18807-38807.test-ssl.local.conf,93%,4.02,13.92,0:19.17,0,27021,17,945164
50001,01.Feb.2013 00:21:13,/usr/sbin/nginx_ensite 38808-88808.test-ssl.local.conf,93%,10.05,35.70,0:49.07,0,67976,17,2288672

nginx_ensite is a small shell script that creates a symlink and does a
nginx -t.


> I don't think that parsing of the config is a culprit. More
> likely it's SSL certificate reading/checking/various random
> initialization/generation. (And may be server names hash
> generation if there are many collisions on server names.)

It's not my experience. In fact, several times I mangled up the cert
names and he generated the not found cert error only quite late in the process.

> Some profiling would be helpful.

I did a ltrace and, hands down, string operations are the main
thing. For a single server block.

awk 'BEGIN {s=0} /strcmp/ {s+=$1} END {print s}' single_ltrace_function_stats_libs.csv
-> 4417

awk 'BEGIN {s=0} /memcpy/ {s+=$1} END {print s}' single_ltrace_function_stats_libs.csv
-> 245

awk 'BEGIN {s=0} /SSL/ {s+=$1} END {print s}' single_ltrace_function_stats_libs.csv
-> 49

So there are 49 ops for SSL functions against 4417 string comparisons.

Is my reasoning flawed?

--- appa





_______________________________________________
nginx-devel mailing list
nginx-devel@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx-devel