Maxim,
You may be a genius.

I see the backlog is only 1024, so I upped that
echo 10024 > /proc/sys/net/ipv4/tcp_max_syn_backlog

Things feel responsive, but I will only know more in 24 hours when we hit
peak loads.

Best,
Khalid

On Mon, Sep 21, 2009 at 9:48 AM, Maxim Dounin <mdounin@mdounin.ru> wrote:

> Hello!
>
> On Mon, Sep 21, 2009 at 09:26:22AM -0700, Khalid Shaikh wrote:
>
> > Team,
> > Here is another sample point. See the 9 second delay on the server I
> keep
> > speaking about. This problem shows across multiple servers. The one we
> are
> > investigating is http://67.159.60.59/status. Same nginx.conf across
> all.
> >
> > root@pa:~# clear; ./status.rb
> > Active connections: 17442
> > server accepts handled requests
> > 291457 291457 499538
> > Reading: 242 Writing: 3838 Waiting: 13362
> > http://208.53.158.42/status 0.07 seconds
> >
> > Active connections: 11545
> > server accepts handled requests
> > 8720635 8720635 13904180
> > Reading: 154 Writing: 1370 Waiting: 10021
> > http://67.159.60.37/status 0.03 seconds
> >
> > Active connections: 3807
> > server accepts handled requests
> > 3758087 3758087 27448407
> > Reading: 44 Writing: 213 Waiting: 3550
> > http://66.90.101.97/status 0.04 seconds
> >
> > Active connections: 8855
> > server accepts handled requests
> > 6865763 6865763 10978021
> > Reading: 133 Writing: 988 Waiting: 7734
> > http://208.53.158.75/status 0.03 seconds
> >
> > Active connections: 10571
> > server accepts handled requests
> > 194174 194174 467424
> > Reading: 106 Writing: 2147 Waiting: 8318
> > http://67.159.60.59/status 9.03 seconds
>
> After looking closely on timings you provide - it's
> suspiciously rounded delays. Looks like packet
> drops/retransmissions. This may be due to some network resource
> shortage on the host in question or some problems on the wire.
>
> First of all, try tcpdump to see what actually happens on the
> wire.
>
> Also it's probably a good idea to examine your backlog size and
> status. AFAIK Linux by default just drops incoming SYN packets on
> listen queue overflows, so this may be the reason.
>
> Maxim Dounin
>
> >
> > Active connections: 8354
> > server accepts handled requests
> > 4314517 4314517 6679147
> > Reading: 195 Writing: 1954 Waiting: 6205
> > http://208.53.158.200/status 0.03 seconds
>
>