Thanks for your response.
Maxim Dounin Wrote:
>
> Are you sure you measure allocated memory?
Yes, I'm looking at VIRT in top rather than RES.
> For
> 16k connections
> openssl (at least 0.9.8, but you claim numbers are
> the same...)
> should allocate about 1.2G (~80k per connection)
> on ssl handshake.
That's odd, because I don't see that. I'm using some slightly hacky code derived from some openssl examples, and here's what it looks like:
/* Build our SSL context*/
ctx=initialize_ctx(KEYFILE,PASSWORD);
int loop;
for (loop = 0; loop < 20000; loop++)
{
printf("Loop %d\n", loop);
/* Connect the TCP socket*/
sock=tcp_connect(host,port);
/* Connect the SSL socket */
ssl=SSL_new(ctx);
sbio=BIO_new_socket(sock,BIO_NOCLOSE);
SSL_set_bio(ssl,sbio,sbio);
if(SSL_connect(ssl)<=0)
berr_exit("SSL connect error");
if(require_server_auth)
check_cert(ssl,host);
/* Now make our HTTP request */
http_request(ssl, loop);
/* Shutdown the socket */
//destroy_ctx(ctx);
//close(sock);
}
The difference in occupancy comes from whether or not I comment out the http_request code. Would you expect the openssl calls above to have incurred the server-side occupancy hit?
>
> Numbers you see in resident memory (RES in top)
> may be quite
> different, but it just means that relevant memory
> wasn't yet
> touched. And grow on request processing is
> expected.
>
Yes, these are easily confused but I'm familiar with the problem.
> > - 16K connections with the outstanding GET
> request now takes ~1GB (down from estimated 1.6GB
> previously)
> >
> > So there's still a massive occupancy cost of
> having an outstanding GET request in the HTTPS
> case, which is surprising. Looks like I'd better
> roll my sleeves up and dig into the code.
>
> For outstanding GET requests there is 16k ssl
> buffer allocated by
> nginx on first output (see
> src/event/ngx_event_openssl.c) and
> freed when connection goes to keepalive state.
> This gives 256M
> for 16k connections.
Ok, that's useful. I'll look at the code, but do you think it's feasible either to
- shrink this (how small could it be? what would happen if it was too small?), or
- free it in between getting a request in and sending the response?
That wouldn't account for all the difference I see, but 256M would go some way to reducing the occupancy hit.
>
> Also there is a bunch of memory in nginx (output
> buffers, large
> client header buffers, gzip buffers, various
> request-related data
> and so on) which are freed when connection goes to
> keepalive
> state. Depending on your settings, particular
> request and already
> sent response this may contribute various numbers
> to keepalive vs.
> outstanding request cases. This should be almost
> the same in
> non-https case though, at least with sendfile not
> used.
Yes, that's what I would expect, and the occupancy in the non-https case is pretty good. So I'm hoping not to have to touch that, and that this is something https-specific.
Edward.
