X All:
I'm attempting to configure nginx to reverse proxy requests from (192.168.0.2:12345) the same Internal Host Address that it's listening from (192.168.0.2:443) on separate ports using the listen and proxy_bind directives.
# /opt/sbin/nginx -v
nginx version: nginx/1.19.2 (x86_64-pc-linux-gnu)
# cat nginx.conf
user admin root;
#user nobody;
worker_processes 1;
events {
worker_connections 64;
}
http {
# HTTPS server
server {
listen 192.168.0.2:443 ssl;
server_name z1.fm;
ssl_certificate /etc/cert.pem;
ssl_certificate_key /etc/key.pem;
proxy_ssl_server_name on;
ssl_session_cache shared:SSL:1m;
ssl_session_timeout 5m;
ssl_ciphers HIGH:!aNULL:!MD5;
ssl_prefer_server_ciphers on;
location / {
# root html;
# index index.html index.htm;
resolver 103.86.99.100;
# proxy_bind 192.168.0.2:12345;
proxy_bind $server_addr:12345;
# proxy_bind $remote_addr:12345 transparent;
proxy_pass $scheme://$host;
}
}
}
I've tried changing the "user admin root;" which is the root user for this router. I've tried using different combinations of "proxy_bind 192.168.0.2;", "proxy_bind 192.168.0.2 transparent;", "proxy_bind $server_addr;", and "proxy_bind $server_addr transparent;". None of them appear to work, when validating with tcpdump. nginx always uses the External WAN Address (100.64.8.236).
Ifconfig Output:
# ifconfig
br0 Link encap:Ethernet HWaddr C0:56:27:D1:B8:A4
inet addr:192.168.0.1 Bcast:192.168.0.255 Mask:255.255.255.0
UP BROADCAST RUNNING ALLMULTI MULTICAST MTU:1500 Metric:1
RX packets:10243803 errors:0 dropped:0 overruns:0 frame:0
TX packets:5440860 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:14614392834 (13.6 GiB) TX bytes:860977246 (821.0 MiB)
br0:0 Link encap:Ethernet HWaddr C0:56:27:D1:B8:A4
inet addr:192.168.0.2 Bcast:192.168.0.255 Mask:255.255.255.0
UP BROADCAST RUNNING ALLMULTI MULTICAST MTU:1500 Metric:1
vlan2 Link encap:Ethernet HWaddr C0:56:27:D1:B8:A4
inet addr:100.64.8.236 Bcast:100.64.15.255 Mask:255.255.248.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:1757588 errors:0 dropped:0 overruns:0 frame:0
TX packets:613625 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:2267961441 (2.1 GiB) TX bytes:139435610 (132.9 MiB)
Route Output:
# route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
10.10.0.17 * 255.255.255.255 UH 0 0 0 tun12
89.38.98.142 100.64.8.1 255.255.255.255 UGH 0 0 0 vlan2
100.64.8.1 * 255.255.255.255 UH 0 0 0 vlan2
10.15.0.65 * 255.255.255.255 UH 0 0 0 tun11
192.168.2.1 * 255.255.255.255 UH 0 0 0 vlan3
51.68.180.4 100.64.8.1 255.255.255.255 UGH 0 0 0 vlan2
192.168.2.0 * 255.255.255.0 U 0 0 0 vlan3
192.168.0.0 * 255.255.255.0 U 0 0 0 br0
100.64.8.0 * 255.255.248.0 U 0 0 0 vlan2
127.0.0.0 * 255.0.0.0 U 0 0 0 lo
default 100.64.8.1 0.0.0.0 UG 0 0 0 vlan2
Tcpdump Output:
Client Remote_Addr (192.168.0.154:$port) == Request => Nginx Reverse Proxy Server - Listener (192.168.0.2:443)
07:19:06.840468 In c8:1f:66:13:a1:11 ethertype IPv4 (0x0800), length 62: 192.168.0.154.55138 > 192.168.0.2.443: Flags [.], ack 1582, win 8212, length 0
07:19:06.840468 In c8:1f:66:13:a1:11 ethertype IPv4 (0x0800), length 62: 192.168.0.154.55138 > 192.168.0.2.443: Flags [.], ack 1582, win 8212, length 0
Nginx Reverse Proxy Server - Listener (192.168.0.2:443) == Response => Client Remote_Addr (192.168.0.154:$port)
07:19:06.841377 Out c0:56:27:d1:b8:a4 ethertype IPv4 (0x0800), length 56: 192.168.0.2.443 > 192.168.0.154.55138: Flags [.], ack 1475, win 541, length 0
07:19:06.841411 Out c0:56:27:d1:b8:a4 ethertype IPv4 (0x0800), length 56: 192.168.0.2.443 > 192.168.0.154.55138: Flags [.], ack 1475, win 541, length 0
Nginx Reverse Proxy Server - Sender (100.64.8.236:12345) == Request => Upstream Desination Server - Listener (104.27.161.206:443)
07:19:11.885314 Out c0:56:27:d1:b8:a4 ethertype IPv4 (0x0800), length 76: 100.64.8.236.12345 > 104.27.161.206.443: Flags [S], seq 3472185855, win 5840, options [mss 1460,sackOK,TS val 331214 ecr 0,nop,wscale 4], length 0
Upstream Desination Server - Listener (104.27.161.206:443) == Response => Nginx Reverse Proxy Server - Sender (100.64.8.236:12345)
07:19:11.887683 In 02:1f:a0:00:00:09 ethertype IPv4 (0x0800), length 68: 104.27.161.206.443 > 100.64.8.236.12345: Flags [S.], seq 2113436779, ack 3472185856, win 65535, options [mss 1400,nop,nop,sackOK,nop,wscale 10], length 0
Note: The Nginx Reverse Proxy Server (Listener) and Nginx Reverse Proxy Server (Sender) MAC addresses are the same piece of hardware
07:19:06.840468 In c8:1f:66:13:a1:11 ethertype IPv4 (0x0800), length 62: 192.168.0.154.55138 > 192.168.0.2.443: Flags [.], ack 1582, win 8212, length 0
07:19:06.840468 In c8:1f:66:13:a1:11 ethertype IPv4 (0x0800), length 62: 192.168.0.154.55138 > 192.168.0.2.443: Flags [.], ack 1582, win 8212, length 0
07:19:06.841377 Out c0:56:27:d1:b8:a4 ethertype IPv4 (0x0800), length 56: 192.168.0.2.443 > 192.168.0.154.55138: Flags [.], ack 1475, win 541, length 0
07:19:06.841411 Out c0:56:27:d1:b8:a4 ethertype IPv4 (0x0800), length 56: 192.168.0.2.443 > 192.168.0.154.55138: Flags [.], ack 1475, win 541, length 0
07:19:11.885314 Out c0:56:27:d1:b8:a4 ethertype IPv4 (0x0800), length 76: 100.64.8.236.12345 > 104.27.161.206.443: Flags [S], seq 3472185855, win 5840, options [mss 1460,sackOK,TS val 331214 ecr 0,nop,wscale 4], length 0
07:19:11.887683 In 02:1f:a0:00:00:09 ethertype IPv4 (0x0800), length 68: 104.27.161.206.443 > 100.64.8.236.12345: Flags [S.], seq 2113436779, ack 3472185856, win 65535, options [mss 1400,nop,nop,sackOK,nop,wscale 10], length 0
07:19:11.887948 Out c0:56:27:d1:b8:a4 ethertype IPv4 (0x0800), length 56: 100.64.8.236.12345 > 104.27.161.206.443: Flags [.], ack 1, win 365, length 0
07:19:11.888854 Out c0:56:27:d1:b8:a4 ethertype IPv4 (0x0800), length 264: 100.64.8.236.12345 > 104.27.161.206.443: Flags [P.], seq 1:209, ack 1, win 365, length 208
07:19:11.890844 In 02:1f:a0:00:00:09 ethertype IPv4 (0x0800), length 62: 104.27.161.206.443 > 100.64.8.236.12345: Flags [.], ack 209, win 66, length 0
07:19:11.893154 In 02:1f:a0:00:00:09 ethertype IPv4 (0x0800), length 1516: 104.27.161.206.443 > 100.64.8.236.12345: Flags [.], seq 1:1461, ack 209, win 66, length 1460
07:19:11.893316 Out c0:56:27:d1:b8:a4 ethertype IPv4 (0x0800), length 56: 100.64.8.236.12345 > 104.27.161.206.443: Flags [.], ack 1461, win 548, length 0
07:19:11.893161 In 02:1f:a0:00:00:09 ethertype IPv4 (0x0800), length 1000: 104.27.161.206.443 > 100.64.8.236.12345: Flags [P.], seq 1461:2405, ack 209, win 66, length 944
Iptables Output:
# iptables -t mangle -I PREROUTING -i vlan2 -p tcp -m multiport --dport 12345 -j MARK --set-mark 0x2000/0x2000
# iptables -t mangle -I POSTROUTING -o vlan2 -p tcp -m multiport --sport 12345 -j MARK --set-mark 0x8000/0x8000
Note: Packets are matching and being marked, but not being routed to the appropriate interfaces. I'm thinking it may be too late in the pipe.
# iptables -t mangle -L -v -n
Chain PREROUTING (policy ACCEPT 5506K packets, 8051M bytes)
pkts bytes target prot opt in out source destination
33 15329 MARK tcp -- vlan2 * 0.0.0.0/0 0.0.0.0/0 multiport dports 12345 MARK or 0x2000
Chain POSTROUTING (policy ACCEPT 2832K packets, 171M bytes)
pkts bytes target prot opt in out source destination
30 4548 MARK tcp -- * vlan2 0.0.0.0/0 0.0.0.0/0 multiport sports 12345 MARK or 0x8000
The reverse proxied requests make it to the destination and back, but using the External WAN Address (100.64.8.236:12345) and not the Internal Host Address (192.168.0.2:12345).
The proxy_bind directive just seems to be failing.
Any ideas?
Thanks!
Gary
