Welcome! Log In Create A New Profile

Advanced

Resolver not re-resolving new ip address of an AW ELB

October 27, 2017 02:58PM
Hello,

We are trying to find a solution from past couple of days but nothing seems to work so far. Our server in prod going down everyday when the AWS ELB changes ip address. Our's little complex proxy (don't ask me why we have to do this :), there is a strong reason for it), our clients will send requests apache (legacy), apache proxies to nginx (new) and nginx decides whether proxy back to apache or serve the request with new micro services. Resolver re-resolves the new micro services (internal alb) ip address but fail to re-resolve the legacy apache (has a ELB with route 53 entry in front). We are using https endpoint to proxy apache request. The request flows thru ELB (legacy) -> Apache -> ELB (new) -> nginx -> ELB (legacy) -> apache

Can you please provide feedback on what are we doing wrong, this is only happening in production. Our load is normal few fundred requests per second. We aren't able to simulate it in test environment.

Here is the configuration:


user nginx;
worker_processes auto;
worker_rlimit_nofile 5120;
error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;

events {
worker_connections 2048;
}

http {
include /etc/nginx/mime.types;
default_type text/plain;

resolver 10....2 172.16.0.23 valid=30s ipv6=off;
resolver_timeout 5s;

log_format main '$proxy_protocol_addr - [$status] - [$request_time] - [$upstream_response_time] - $server_name $upstream_addr $request';

access_log /var/log/nginx/error.log main;

rewrite_log on;

client_body_timeout 60s;
client_header_timeout 30s;
send_timeout 60s;
sendfile off;

tcp_nodelay on;
tcp_nopush on;
reset_timedout_connection on;

server_names_hash_bucket_size 128;
client_body_buffer_size 64k;
client_max_body_size 10m;

server {

listen 443 ssl proxy_protocol default_server;
server_name mydomain.com;
ssl_certificate mydomain.crt;
ssl_certificate_key mydomain.key;

set $alb_upstream aws-internal-alb;
set $apache_upstream legacy.domain.com;

proxy_buffers 8 24k;
proxy_buffer_size 2k;
proxy_http_version 1.1;
proxy_set_header Connection "";
proxy_set_header X-Real-IP $proxy_protocol_addr;
proxy_next_upstream off;


location /services/(migrated1|migrated2)/ {

proxy_set_header Host $host;
proxy_connect_timeout 302;
proxy_read_timeout 302;

rewrite /services/(.*) /$1?$args break;
proxy_pass http://alb_upstream;
}

location /services/ {
proxy_set_header x-nginx-rejected true;
proxy_set_header Host legacy.domain.com;
proxy_connect_timeout 302;
proxy_read_timeout 302;

rewrite /services/(.*) /$1?$args break;
proxy_pass https://$apache_upstream;
}

}
}

Thanks in advance.
RK
Subject Author Posted

Resolver not re-resolving new ip address of an AW ELB

RKGood October 27, 2017 02:58PM

Re: Resolver not re-resolving new ip address of an AW ELB

RKGood October 27, 2017 03:00PM

Re: Resolver not re-resolving new ip address of an AW ELB

RKGood October 30, 2017 11:54AM

Re: Resolver not re-resolving new ip address of an AW ELB

Nishikubo Minoru October 30, 2017 09:14PM

Re: Resolver not re-resolving new ip address of an AW ELB

pankaj@releasemanager.in October 30, 2017 11:25PM

Re: Resolver not re-resolving new ip address of an AW ELB

RKGood October 31, 2017 01:26PM

Re: Resolver not re-resolving new ip address of an AW ELB

ru@nginx.com November 02, 2017 06:06AM



Sorry, only registered users may post in this forum.

Click here to login

Online Users

Guests: 68
Record Number of Users: 6 on February 13, 2018
Record Number of Guests: 421 on December 02, 2018
Powered by nginx      Powered by FreeBSD      PHP Powered      Powered by MariaDB      ipv6 ready