Hello,
We are using NGINX stream TCP proxy with the proxy_bind transparent directive for IP transparency (https://www.f5.com/company/blog/nginx/ip-transparency-direct-server-return-nginx-plus-transparent-proxy). We have observed that if we don't use proxy_bind transparent to spoof the client IP, everything works fine. We use this setup as a kind of CDN to inspect packets for the CDN and route them to another load balancer (LB) by SNI.
There is an issue that this load balancer cannot work with the PROXY protocol, and we don't have the client IP address in the packet. So, we use this method to spoof the client address with proxy_bind $remote_addr transparent;. Now the IP in the logs is real, but during testing, we discovered that there are issues with connection stability. When we try to download a 4GB file, we observe connection drops and retransmissions. The download speed peaks, but when we don't use the proxy_bind directive, everything is good. We also tried using a non-local IP address with proxy_bind set to the address of the NGINX instance, but the problems remain the same.
nginx.conf:
user root;
worker_processes 16;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;
worker_rlimit_nofile 65535;
events {
# determines how much clients will be served per worker
# max clients = worker_connections * worker_processes
# max clients is also limited by the number of socket connections available on the system (~64k)
worker_connections 65535;
# optimized to serve many clients with each thread, essential for linux -- for testing environment
#use epoll;
# accept as many connections as possible, may flood worker connections if set too low -- for testing environment
multi_accept on;
}
http {
#modsecurity on;
#modsecurity_rules_file /etc/nginx/modsecurity.conf;
#Zabbix monitoring
server {
listen 127.0.0.1:80 default_server;
location = /basic_status {
stub_status;
allow 127.0.0.1;
allow ::1;
deny all;
}}
#Drop without hostname
server {
listen 80 default_server;
return 444;
}
#Test client
server {
ssl_certificate /etc/ssl/certs/nginx-selfsigned.crt;
ssl_certificate_key /etc/ssl/private/nginx-selfsigned.key;
listen 4431 ssl;
server_name <redacted>;
location / {
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_pass https://10.21.37.1:443;
}
}
##
# Basic Settings
##
keepalive_timeout 65;
sendfile on;
tcp_nopush on;
types_hash_max_size 2048;
# server_tokens off;
# server_names_hash_bucket_size 64;
# server_name_in_redirect off;
include /etc/nginx/mime.types;
default_type application/octet-stream;
##
# SSL Settings
##
ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3; # Dropping SSLv3, ref: POODLE
ssl_prefer_server_ciphers on;
##
# Logging Settings
##
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log crit;
##
# Gzip Settings
##
gzip on;
# gzip_vary on;
# gzip_proxied any;
# gzip_comp_level 6;
# gzip_buffers 16 8k;
# gzip_http_version 1.1;
# gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;
##
# Virtual Host Configs
##
include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;
}
#mail {
# # See sample authentication script at:
# # http://wiki.nginx.org/ImapAuthenticateWithApachePhpScript
#
# # auth_http localhost/auth.php;
# # pop3_capabilities "TOP" "USER";
# # imap_capabilities "IMAP4rev1" "UIDPLUS";
#
# server {
# listen localhost:110;
# protocol pop3;
# proxy on;
# }
#
# server {
# listen localhost:143;
# protocol imap;
# proxy on;
# }
#}
include /etc/nginx/passthrough.conf;
passthrough.conf:
stream {
map $ssl_preread_server_name $name {
<redacted> <redacted>_stream;
<redacted> <redacted>_stream;
}
upstream <redacted>_stream {
server 172.16.0.10:443;
}
upstream <redacted>_stream {
server 10.21.37.1:443;
}
server {
listen 443;
proxy_pass $name;
ssl_preread on;
#proxy_protocol on;
proxy_bind $remote_addr transparent;
resolver <redacted>;
}
log_format basic '$remote_addr [$time_local] '
'$protocol $status $bytes_sent $bytes_received '
'$session_time "$upstream_addr" '
'"$upstream_bytes_sent" "$upstream_bytes_received" "$upstream_connect_time"';
# access_log /var/log/nginx/stream.access.log basic;
# error_log /var/log/nginx/stream.error.log debug;
}
Download with proxybind:
Length: 4613734400 (4,3G) [application/octet-stream]
Saving to: ‘/dev/null’
/dev/null 14%[=> ] 639,72M 320MB/s
/dev/null 14%[=> ] 639,72M 131MB/s
/dev/null 14%[=> ] 639,72M --.-KB/s
/dev/null 14%[=> ] 639,72M --.-KB/s
/dev/null 14%[=> ] 639,72M --.-KB/s
2024-07-18 09:08:10 (119 MB/s) - Read error at byte 2908438224/4613734400 (The TLS connection was non-properly terminated.). Retrying.
Download without proxybind:
HTTP request sent, awaiting response... 200 OK
Length: 4613734400 (4,3G) [application/octet-stream]
Saving to: ‘/dev/null’
/dev/null 100%[===================>] 4,30G 330MB/s in 14s
2024-07-18 09:09:41 (324 MB/s) - ‘/dev/null’ saved [4613734400/4613734400]
We managed to switch many config options, but there was no real benefit, also we tried to tune linux kernel - also without any success.