Welcome! Log In Create A New Profile

Advanced

Re: Nginx throttling issue?

All files from this thread

File Name File Size   Posted by Date  
image003.jpg 3.2 KB open | download John Melom 03/26/2018 Read message
rawRespScatterplot.png 42.3 KB open | download John Melom 03/26/2018 Read message
April 05, 2018 12:48AM
John,

I think that you need to understand what is happening on your host throughout the duration of the test. Specifically, what is happening with the tcp connections. If you run netstat and grep for tcp and do this in a loop every say five seconds then you’ll see how many connections peak get created.
If the thing you are testing exists in production then you are lucky. You can do the same in production and see what it is that you need to replicate.

You didn’t mention whether you had persistent connections (http keep alive) configured. This is key to maximizing scalability. You did say that you were using SSL. If it were me I’d use a load generator that more closely resembles the behavior of real users on a website. Wrk2, Tsung, httperf, Gatling are examples of some that do. Using jmeter with zero think time is a very common anti pattern that doesn’t behave anything like real users. I think of it as the lazy performance tester pattern.

Imagine a real web server under heavy load from human beings. You will see thousands of concurrent connections but fewer concurrent requests in flight. With the jmeter zero think time model then you are either creating new connections or reusing them - so either you have a shitload of connections and your nginx process starts running out of file handles or you are jamming requests down a single connection- neither of which resemble reality.

If you are committed to using jmeter for some reason then use more instances with real thinktimes. Each instance’s connection wil have a different source port

Sent from my iPhone

> On Apr 4, 2018, at 5:20 PM, John Melom <John.Melom@spok.com> wrote:
>
> Hi Maxim,
>
> I've looked at the nstat data and found the following values for counters:
>
>> nstat -az | grep -I listen
> TcpExtListenOverflows 0 0.0
> TcpExtListenDrops 0 0.0
> TcpExtTCPFastOpenListenOverflow 0 0.0
>
>
> nstat -az | grep -i retra
> TcpRetransSegs 12157 0.0
> TcpExtTCPLostRetransmit 0 0.0
> TcpExtTCPFastRetrans 270 0.0
> TcpExtTCPForwardRetrans 11 0.0
> TcpExtTCPSlowStartRetrans 0 0.0
> TcpExtTCPRetransFail 0 0.0
> TcpExtTCPSynRetrans 25 0.0
>
> Assuming the above "Listen" counters provide data about the overflow issue you mention, then there are no overflows on my system. While retransmissions are happening, it doesn't seem they are related to listen queue overflows.
>
>
> Am I looking at the correct data items? Is my interpretation of the data correct? If so, do you have any other ideas I could investigate?
>
> Thanks,
>
> John
>
> -----Original Message-----
> From: nginx [mailto:nginx-bounces@nginx.org] On Behalf Of John Melom
> Sent: Tuesday, March 27, 2018 8:52 AM
> To: nginx@nginx.org
> Subject: RE: Nginx throttling issue?
>
> Maxim,
>
> Thank you for your reply. I will look to see if "netstat -s" detects any listen queue overflows.
>
> John
>
>
> -----Original Message-----
> From: nginx [mailto:nginx-bounces@nginx.org] On Behalf Of Maxim Dounin
> Sent: Tuesday, March 27, 2018 6:55 AM
> To: nginx@nginx.org
> Subject: Re: Nginx throttling issue?
>
> Hello!
>
>> On Mon, Mar 26, 2018 at 08:21:27PM +0000, John Melom wrote:
>>
>> I am load testing our system using Jmeter as a load generator.
>> We execute a script consisting of an https request executing in a
>> loop. The loop does not contain a think time, since at this point I
>> am not trying to emulate a “real user”. I want to get a quick look at
>> our system capacity. Load on our system is increased by increasing
>> the number of Jmeter threads executing our script. Each Jmeter thread
>> references different data.
>>
>> Our system is in AWS with an ELB fronting Nginx, which serves as a
>> reverse proxy for our Docker Swarm application cluster.
>>
>> At moderate loads, a subset of our https requests start experiencing
>> to a 1 second delay in addition to their normal response time. The
>> delay is not due to resource contention.
>> System utilizations remain low. The response times cluster around 4
>> values: 0 millilseconds, 50 milliseconds, 1 second, and 1.050
>> seconds. Right now, I am most interested in understanding and
>> eliminating the 1 second delay that gives the clusters at 1 second and
>> 1.050 seconds.
>>
>> The attachment shows a response time scatterplot from one of our runs.
>> The x-axis is the number of seconds into the run, the y-axis is the
>> response time in milliseconds. The plotted data shows the response
>> time of requests at the time they occurred in the run.
>>
>> If I run the test bypassing the ELB and Nginx, this delay does not
>> occur.
>> If I bypass the ELB, but include Nginx in the request path, the delay
>> returns.
>>
>> This leads me to believe the 1 second delay is coming from Nginx.
>
> There are no magic 1 second delays in nginx - unless you've configured something explicitly.
>
> Most likely, the 1 second delay is coming from TCP retransmission timeout during connection establishment due to listen queue overflows. Check "netstat -s" to see if there are any listen queue overflows on your hosts.
>
> [...]
>
> --
> Maxim Dounin
> http://mdounin.ru/
> _______________________________________________
> nginx mailing list
> nginx@nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx
>
> ________________________________
> NOTE: This email message and any attachments are for the sole use of the intended recipient(s) and may contain confidential and/or privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you have received this e-mail in error, please contact the sender by replying to this email, and destroy all copies of the original message and any material included with this email.
> _______________________________________________
> nginx mailing list
> nginx@nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx
>
> ________________________________
> NOTE: This email message and any attachments are for the sole use of the intended recipient(s) and may contain confidential and/or privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you have received this e-mail in error, please contact the sender by replying to this email, and destroy all copies of the original message and any material included with this email.
> _______________________________________________
> nginx mailing list
> nginx@nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Subject Author Posted

Nginx throttling issue? Attachments

John Melom March 26, 2018 04:22PM

Re: Nginx throttling issue?

pbooth March 26, 2018 04:58PM

RE: Nginx throttling issue?

John Melom March 27, 2018 09:50AM

Re: Nginx throttling issue?

Maxim Dounin March 27, 2018 07:56AM

RE: Nginx throttling issue?

John Melom March 27, 2018 09:54AM

RE: Nginx throttling issue?

John Melom April 04, 2018 05:22PM

Re: Nginx throttling issue?

pbooth April 05, 2018 12:48AM

Re: Nginx throttling issue?

Richard Stanway via nginx April 06, 2018 01:14PM

Re: Nginx throttling issue?

Maxim Dounin April 09, 2018 09:32AM



Sorry, only registered users may post in this forum.

Click here to login

Online Users

Guests: 165
Record Number of Users: 8 on April 13, 2023
Record Number of Guests: 500 on July 15, 2024
Powered by nginx      Powered by FreeBSD      PHP Powered      Powered by MariaDB      ipv6 ready