Welcome! Log In Create A New Profile

Advanced

TCPv6: Possible SYN flooding on port 80. Sending cookies.

Posted by omkar_jadhav_20 
TCPv6: Possible SYN flooding on port 80. Sending cookies.
September 21, 2016 03:22AM
Hi,
I am using nginx with version 1.4.0. Getting error as : TCPv6: Possible SYN flooding on port 80. Sending cookies.
It is serving media traffic and whenever number of incoming connections crosses 7K. file size served is about 1.5Mb.
below are few details for reference :

part of nginx conf :
worker_processes auto;
events {
worker_connections 4096;
use epoll;
multi_accept on;
}
worker_rlimit_nofile 100001;

http {
include mime.types;
default_type video/mp4;
proxy_buffering on;
proxy_buffer_size 4096k;
proxy_buffers 5 4096k;
sendfile on;
keepalive_timeout 30;
tcp_nodelay on;
tcp_nopush on;
reset_timedout_connection on;
gzip off;
server_tokens off;


sysctl parameters are :
net.core.somaxconn=65535
net.ipv4.tcp_tw_recycle = 0 # purpsely set to 1 as NATed client were facing issue.
net.ipv4.tcp_syncookies = 1
net.core.netdev_max_backlog = 250000
net.ipv4.tcp_max_syn_backlog = 4096
net.core.rmem_max = 134217728
net.core.wmem_max = 134217728
net.ipv4.tcp_rmem = 4096 87380 67108864
net.ipv4.tcp_wmem = 4096 65536 67108864
net.core.netdev_max_backlog = 250000
net.ipv4.tcp_congestion_control = htcp
net.ipv4.tcp_mtu_probing = 1
net.ipv4.ip_local_port_range = 4096 65535
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_fin_timeout = 10

server used : CentOS release 6.5 (Final)

# cat /proc/meminfo
MemTotal: 1588152040 kB

# lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 60
Re: TCPv6: Possible SYN flooding on port 80. Sending cookies.
September 23, 2016 01:43AM
Can someone please assist here
Re: TCPv6: Possible SYN flooding on port 80. Sending cookies.
September 28, 2016 06:25AM
please assist in this case
Re: TCPv6: Possible SYN flooding on port 80. Sending cookies.
September 28, 2016 08:10AM
At least upgrade to the latest version of nginx.

---
nginx for Windows http://nginx-win.ecsds.eu/
Re: TCPv6: Possible SYN flooding on port 80. Sending cookies.
September 29, 2016 03:42AM
Hi itpp2012 ,
I have upgraded nginx on one of our setup , load balancer which is haproxy is equally distributing load on user facing servers but one of the user facing server's writing requests are more and hence resposne time for tht server is increasing as number of requests gets increased in peak time.
PFA screenshots of user facing servers.Please assist
Attachments:
open | download - user_facing_server.png (5.7 KB)
open | download - user_facing_server_morewriting.png (5.9 KB)
Re: TCPv6: Possible SYN flooding on port 80. Sending cookies.
September 29, 2016 03:53AM
Try:

worker_connections 16384;
multi_accept off;

Also you have set
'net.ipv4.tcp' values, but what are your ipv6 values?

---
nginx for Windows http://nginx-win.ecsds.eu/
Re: TCPv6: Possible SYN flooding on port 80. Sending cookies.
September 29, 2016 04:46AM
forget to mention that I have updated nginx to nginx/1.10.1. We can not make multi_accept off; as our application requires nginx to make a worker process accept all new connections at a time. Also attached are the list of ipv6 sysctl parameter for this server. Please let me know in case something needs to be modified. I have changed worker_connections 16384 and keeping the server under observation for some time (at least 20 mins).
Attachments:
open | download - sysctlipv6 (26.6 KB)
Re: TCPv6: Possible SYN flooding on port 80. Sending cookies.
September 29, 2016 04:58AM
below is the nginx status on both server , server who was having issue with number of writing was now observed with still increase in writing requests :
serer without any issue :
Active connections: 7045
server accepts handled requests
207550 207550 238713
Reading: 0 Writing: 726 Waiting: 6319

with issue :
Active connections: 9927
server accepts handled requests
277077 277077 317874
Reading: 0 Writing: 4152 Waiting: 5775

PFB part of nginx.conf of server with issue :
worker_processes auto;
events {
worker_connections 4096;
use epoll;
multi_accept on;
}
worker_rlimit_nofile 100001;

http {
include mime.types;
default_type video/mp4;
proxy_buffering on;
proxy_buffer_size 4096k;
proxy_buffers 5 4096k;
sendfile on;
keepalive_timeout 30;
keepalive_requests 60000;
send_timeout 10;
tcp_nodelay on;
tcp_nopush on;
reset_timedout_connection on;
gzip off;
server_tokens off;
#aio threads;
Re: TCPv6: Possible SYN flooding on port 80. Sending cookies.
September 29, 2016 05:21AM
multi_accept is known to do this kind of behavior.
Try without epoll (which is also known to cause congestion)

---
nginx for Windows http://nginx-win.ecsds.eu/
Re: TCPv6: Possible SYN flooding on port 80. Sending cookies.
September 29, 2016 05:44AM
Hi,
We can not change epoll as our server os is of centos and it was recommended to keep epoll yes for rhel. Number of requests coming on user facing servers are very huge in our environment. Could you let us know what should be ideal value of vm.swappiness. we have vm.swappiness=60 which is default one but red hat recommends it to be 10. Please suggest let me know in case you require any details from me related to nginx/system.
Re: TCPv6: Possible SYN flooding on port 80. Sending cookies.
September 29, 2016 07:07AM
If you are not willing to change settings to find out what is causing this I can't help you.

---
nginx for Windows http://nginx-win.ecsds.eu/
Re: TCPv6: Possible SYN flooding on port 80. Sending cookies.
September 29, 2016 07:13AM
Ok let me check without epoll..
Re: TCPv6: Possible SYN flooding on port 80. Sending cookies.
September 29, 2016 07:23AM
Done on both user facing servers but no improvement :
nginx_stats of server with more writing req :

Active connections: 7273
server accepts handled requests
221000 221000 257718
Reading: 0 Writing: 2599 Waiting: 4674

nginx_stats of server with less writing req:
Active connections: 5838
server accepts handled requests
162471 162471 190024
Reading: 0 Writing: 682 Waiting: 5156

Does any sysctl parameter causing this issue or any other config ? both the servers are exactly running on same os with same patch levels/kernel version.



Edited 1 time(s). Last edit at 09/29/2016 07:24AM by omkar_jadhav_20.
Re: TCPv6: Possible SYN flooding on port 80. Sending cookies.
September 29, 2016 08:17AM
Possible reasons:
http://stackoverflow.com/questions/28823254/too-many-nginx-waiting-connections
https://forum.nginx.org/read.php?2,49182,49196#msg-49196
https://forum.nginx.org/read.php?2,261632,261664#msg-261664

---
nginx for Windows http://nginx-win.ecsds.eu/
Sorry, only registered users may post in this forum.

Click here to login

Online Users

Guests: 176
Record Number of Users: 8 on April 13, 2023
Record Number of Guests: 421 on December 02, 2018
Powered by nginx      Powered by FreeBSD      PHP Powered      Powered by MariaDB      ipv6 ready