Welcome! Log In Create A New Profile

Advanced

Re: Occupancy of SSL connections?

Maxim Dounin
November 12, 2009 07:28PM
Hello!

On Thu, Nov 12, 2009 at 01:11:08PM -0500, edwh2 wrote:

> Thanks for your response.
>
> Maxim Dounin Wrote:
>
> >
> > Are you sure you measure allocated memory?
>
> Yes, I'm looking at VIRT in top rather than RES.
>
> > For
> > 16k connections
> > openssl (at least 0.9.8, but you claim numbers are
> > the same...)
> > should allocate about 1.2G (~80k per connection)
> > on ssl handshake.
>
> That's odd, because I don't see that. I'm using some slightly hacky code derived from some openssl examples, and here's what it looks like:
>
> /* Build our SSL context*/
> ctx=initialize_ctx(KEYFILE,PASSWORD);
>
> int loop;
> for (loop = 0; loop < 20000; loop++)
> {
> printf("Loop %d\n", loop);
> /* Connect the TCP socket*/
> sock=tcp_connect(host,port);
>
> /* Connect the SSL socket */
> ssl=SSL_new(ctx);
> sbio=BIO_new_socket(sock,BIO_NOCLOSE);
> SSL_set_bio(ssl,sbio,sbio);
>
> if(SSL_connect(ssl)<=0)
> berr_exit("SSL connect error");
> if(require_server_auth)
> check_cert(ssl,host);
>
> /* Now make our HTTP request */
> http_request(ssl, loop);
>
> /* Shutdown the socket */
> //destroy_ctx(ctx);
> //close(sock);
> }
>
> The difference in occupancy comes from whether or not I comment out the http_request code. Would you expect the openssl calls above to have incurred the server-side occupancy hit?

See nothing obviously wrong here, but can't say anything
for sure without full code.

Looking into openssl source, there are at least the following
allocation right after it got ClientHello message (ssl3/tls1
case):

~21k in ssl3_accept() for s->init_buf

~34k in ssl3_setup_buffers() for s->s3->rbuf (as nginx uses
SSL_OP_MICROSOFT_BIG_SSLV3_BUFFER workaround option)

~18k in ssl3_setup_buffers() for s->s3->wbuf

About ~70k in total.

Also there are about 8k for input/ouput bio buffers as well, and
various small allocations for state information (including copy of
server cert and so on). This looks much closer to 270M per 16k
connections you see, so probably your client doesn't send
ClientHello for some reason.

You may try running nginx with debug logging enabled and take a
look if ssl handshake actually happens. There should be '[debug] ...
SSL: ..., cipher: ..." lines right after ssl handshake completed.
Also you may try tcpdump to see what happens on the wire.

[...]

> > For outstanding GET requests there is 16k ssl
> > buffer allocated by
> > nginx on first output (see
> > src/event/ngx_event_openssl.c) and
> > freed when connection goes to keepalive state.
> > This gives 256M
> > for 16k connections.
>
> Ok, that's useful. I'll look at the code, but do you think it's feasible either to
> - shrink this (how small could it be? what would happen if it was too small?), or
> - free it in between getting a request in and sending the response?

It's already allocated only when needed (i.e. when nginx starts
sending the response, and freed when connection goes to keepalive
state - i.e. after sending the response). Reducing it will likely
render it useless - it's here to reduce ssl overhead by feeding
openssl with big chunks of data instead multiple small ones.

Anyway - it's almost nothing compared to openssl's own buffers.

Maxim Dounin
Subject Author Posted

Occupancy of SSL connections?

Edward Hibbert November 09, 2009 12:36PM

RE: Occupancy of SSL connections?

Edward Hibbert November 09, 2009 12:40PM

Re: Occupancy of SSL connections?

Igor Sysoev November 09, 2009 02:44PM

Re: Occupancy of SSL connections?

edwh2 November 11, 2009 06:03AM

Re: Occupancy of SSL connections?

edwh2 November 12, 2009 09:15AM

Re: Occupancy of SSL connections?

Maxim Dounin November 12, 2009 10:54AM

Re: Occupancy of SSL connections?

edwh2 November 12, 2009 01:11PM

Re: Occupancy of SSL connections?

Maxim Dounin November 12, 2009 07:28PM

Re: Occupancy of SSL connections?

Cliff Wells November 09, 2009 01:10PM

Re: Occupancy of SSL connections?

Cliff Wells November 09, 2009 01:18PM

Re: Occupancy of SSL connections?

edwh2 November 11, 2009 06:18AM



Sorry, only registered users may post in this forum.

Click here to login

Online Users

Guests: 127
Record Number of Users: 8 on April 13, 2023
Record Number of Guests: 500 on July 15, 2024
Powered by nginx      Powered by FreeBSD      PHP Powered      Powered by MariaDB      ipv6 ready