Hello!

On Thu, Nov 12, 2009 at 09:15:14AM -0500, edwh2 wrote:

> -------------------------------------------------------
> > On Mon, Nov 09, 2009 at 05:34:12PM +0000, Edward
> > Hibbert wrote:
> >
> > > For HTTP connections, I can get about 20K
> > connections and outstanding requests in 116MB,
> > which is pretty respectable.
> > >
> > > However with HTTPS, what I see is that:
> > > - having the connection open takes 261MB for
> > about 16K connections
> > > - having the connection open with an outstanding
> > request takes about 1GB for 10L connections.
>
> Using the OpenSSL 1.0 (thanks for the suggestion, Cliff) there's a significant drop in occupancy.
> - 16K open connections now takes ~277MB (noise level difference from previous measurement)

Are you sure you measure allocated memory? For 16k connections
openssl (at least 0.9.8, but you claim numbers are the same...)
should allocate about 1.2G (~80k per connection) on ssl handshake.

Numbers you see in resident memory (RES in top) may be quite
different, but it just means that relevant memory wasn't yet
touched. And grow on request processing is expected.

> - 16K connections with the outstanding GET request now takes ~1GB (down from estimated 1.6GB previously)
>
> So there's still a massive occupancy cost of having an outstanding GET request in the HTTPS case, which is surprising. Looks like I'd better roll my sleeves up and dig into the code.

For outstanding GET requests there is 16k ssl buffer allocated by
nginx on first output (see src/event/ngx_event_openssl.c) and
freed when connection goes to keepalive state. This gives 256M
for 16k connections.

Also there is a bunch of memory in nginx (output buffers, large
client header buffers, gzip buffers, various request-related data
and so on) which are freed when connection goes to keepalive
state. Depending on your settings, particular request and already
sent response this may contribute various numbers to keepalive vs.
outstanding request cases. This should be almost the same in
non-https case though, at least with sendfile not used.

Maxim Dounin