Hello,

We are trying to find a solution from past couple of days but nothing seems to work so far. Our server in prod going down everyday when the AWS ELB changes ip address. Our's little complex proxy (don't ask me why we have to do this :), there is a strong reason for it), our clients will send requests apache (legacy), apache proxies to nginx (new) and nginx decides whether proxy back to apache or serve the request with new micro services. Resolver re-resolves the new micro services (internal alb) ip address but fail to re-resolve the legacy apache (has a ELB with route 53 entry in front). We are using https endpoint to proxy apache request. The request flows thru ELB (legacy) -> Apache -> ELB (new) -> nginx -> ELB (legacy) -> apache

Can you please provide feedback on what are we doing wrong, this is only happening in production. Our load is normal few fundred requests per second. We aren't able to simulate it in test environment.

Here is the configuration:


user nginx;
worker_processes auto;
worker_rlimit_nofile 5120;
error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;

events {
worker_connections 2048;
}

http {
include /etc/nginx/mime.types;
default_type text/plain;

resolver 10....2 172.16.0.23 valid=30s ipv6=off;
resolver_timeout 5s;

log_format main '$proxy_protocol_addr - [$status] - [$request_time] - [$upstream_response_time] - $server_name $upstream_addr $request';

access_log /var/log/nginx/error.log main;

rewrite_log on;

client_body_timeout 60s;
client_header_timeout 30s;
send_timeout 60s;
sendfile off;

tcp_nodelay on;
tcp_nopush on;
reset_timedout_connection on;

server_names_hash_bucket_size 128;
client_body_buffer_size 64k;
client_max_body_size 10m;

server {

listen 443 ssl proxy_protocol default_server;
server_name mydomain.com;
ssl_certificate mydomain.crt;
ssl_certificate_key mydomain.key;

set $alb_upstream aws-internal-alb;
set $apache_upstream legacy.domain.com;

proxy_buffers 8 24k;
proxy_buffer_size 2k;
proxy_http_version 1.1;
proxy_set_header Connection "";
proxy_set_header X-Real-IP $proxy_protocol_addr;
proxy_next_upstream off;


location /services/(migrated1|migrated2)/ {

proxy_set_header Host $host;
proxy_connect_timeout 302;
proxy_read_timeout 302;

rewrite /services/(.*) /$1?$args break;
proxy_pass http://alb_upstream;
}

location /services/ {
proxy_set_header x-nginx-rejected true;
proxy_set_header Host legacy.domain.com;
proxy_connect_timeout 302;
proxy_read_timeout 302;

rewrite /services/(.*) /$1?$args break;
proxy_pass https://$apache_upstream;
}

}
}

Thanks in advance.
RK