November 12, 2009 01:11PM
Thanks for your response.

Maxim Dounin Wrote:

>
> Are you sure you measure allocated memory?

Yes, I'm looking at VIRT in top rather than RES.

> For
> 16k connections
> openssl (at least 0.9.8, but you claim numbers are
> the same...)
> should allocate about 1.2G (~80k per connection)
> on ssl handshake.

That's odd, because I don't see that. I'm using some slightly hacky code derived from some openssl examples, and here's what it looks like:

/* Build our SSL context*/
ctx=initialize_ctx(KEYFILE,PASSWORD);

int loop;
for (loop = 0; loop < 20000; loop++)
{
printf("Loop %d\n", loop);
/* Connect the TCP socket*/
sock=tcp_connect(host,port);

/* Connect the SSL socket */
ssl=SSL_new(ctx);
sbio=BIO_new_socket(sock,BIO_NOCLOSE);
SSL_set_bio(ssl,sbio,sbio);

if(SSL_connect(ssl)<=0)
berr_exit("SSL connect error");
if(require_server_auth)
check_cert(ssl,host);

/* Now make our HTTP request */
http_request(ssl, loop);

/* Shutdown the socket */
//destroy_ctx(ctx);
//close(sock);
}

The difference in occupancy comes from whether or not I comment out the http_request code. Would you expect the openssl calls above to have incurred the server-side occupancy hit?

>
> Numbers you see in resident memory (RES in top)
> may be quite
> different, but it just means that relevant memory
> wasn't yet
> touched. And grow on request processing is
> expected.
>

Yes, these are easily confused but I'm familiar with the problem.

> > - 16K connections with the outstanding GET
> request now takes ~1GB (down from estimated 1.6GB
> previously)
> >
> > So there's still a massive occupancy cost of
> having an outstanding GET request in the HTTPS
> case, which is surprising. Looks like I'd better
> roll my sleeves up and dig into the code.
>
> For outstanding GET requests there is 16k ssl
> buffer allocated by
> nginx on first output (see
> src/event/ngx_event_openssl.c) and
> freed when connection goes to keepalive state.
> This gives 256M
> for 16k connections.

Ok, that's useful. I'll look at the code, but do you think it's feasible either to
- shrink this (how small could it be? what would happen if it was too small?), or
- free it in between getting a request in and sending the response?

That wouldn't account for all the difference I see, but 256M would go some way to reducing the occupancy hit.

>
> Also there is a bunch of memory in nginx (output
> buffers, large
> client header buffers, gzip buffers, various
> request-related data
> and so on) which are freed when connection goes to
> keepalive
> state. Depending on your settings, particular
> request and already
> sent response this may contribute various numbers
> to keepalive vs.
> outstanding request cases. This should be almost
> the same in
> non-https case though, at least with sendfile not
> used.

Yes, that's what I would expect, and the occupancy in the non-https case is pretty good. So I'm hoping not to have to touch that, and that this is something https-specific.

Edward.
Subject Author Posted

Occupancy of SSL connections?

Edward Hibbert November 09, 2009 12:36PM

RE: Occupancy of SSL connections?

Edward Hibbert November 09, 2009 12:40PM

Re: Occupancy of SSL connections?

Igor Sysoev November 09, 2009 02:44PM

Re: Occupancy of SSL connections?

edwh2 November 11, 2009 06:03AM

Re: Occupancy of SSL connections?

edwh2 November 12, 2009 09:15AM

Re: Occupancy of SSL connections?

Maxim Dounin November 12, 2009 10:54AM

Re: Occupancy of SSL connections?

edwh2 November 12, 2009 01:11PM

Re: Occupancy of SSL connections?

Maxim Dounin November 12, 2009 07:28PM

Re: Occupancy of SSL connections?

Cliff Wells November 09, 2009 01:10PM

Re: Occupancy of SSL connections?

Cliff Wells November 09, 2009 01:18PM

Re: Occupancy of SSL connections?

edwh2 November 11, 2009 06:18AM



Sorry, only registered users may post in this forum.

Click here to login

Online Users

Guests: 180
Record Number of Users: 8 on April 13, 2023
Record Number of Guests: 500 on July 15, 2024
Powered by nginx      Powered by FreeBSD      PHP Powered      Powered by MariaDB      ipv6 ready