Welcome! Log In Create A New Profile

Advanced

Issue with NGINX as reverse proxy for grpc service

Víctor Enríquez
August 07, 2020 10:20AM
Hi,

So we have a service exposing a grpc interface under a certain location
and we are using nginx in front of it. The config looks like the
following:

upstream grpcservers {
server fqdn:port;
server fqdn:port;
}

....

server {
listen port ssl http2;
client_max_body_size 15m;
server_name fqdn;

ssl_certificate /etc/certs/server.crt;
ssl_certificate_key /etc/certs/server.key;

location /my.location. {
grpc_set_header X-Ip-Address $remote_addr;
grpc_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
grpc_ssl_certificate /etc/ssl/mtls-client.crt;
grpc_ssl_certificate_key /etc/ssl/mtls-client.key;
grpc_pass grpcs://grpcservers;
...
}

# Error responses
include conf.d/errors.grpc_conf; # gRPC-compliant error responses
default_type application/grpc; # Ensure gRPC for all error
responses

} //End of the server directive

Now we just realized that each time we do a GET / to that specific port
under that specific location using curl --http2, the request is
forwarded to the backend in such a way that it makes nginx believe that
the backend has crashed, allowing anyone to DDoS this particular
service by just repeteadly sending GET / request to the endpoint.

I am seeing the following messages in the logs:

020/08/07 13:02:37 [error] 1100#1100: *199 upstream rejected request
with error 2 while reading response header from upstream, client:
X.X.X.X, server: fqdn1, request: "POST /my.location.magic.API/GetMagic
HTTP/2.0", upstream: "grpcs://Z.Z.Z.Z:PORT", host: "fqdn1:PORT"

Eventually after the 2 upstream servers are marked as failed we get the
following message:

020/08/07 11:07:05 [error] 1100#1100: *96 no live upstreams while
connecting to upstream, client: X.X.X.X, server: fqdn1, request: "POST
/my.location.magic.API/GetMagic HTTP/2.0", upstream:
"grpcs://grpcservers", host: "fqdn1:PORT"

until the servers are marked as valid again. And the cycle repeats.

I am not an expert on HTTP/2 or gRPC, but it seems like nginx is unable
to negotiate the connection with the backend for those particular
requests created with curl and ends marking the backend as failed when
in fact, it is not failing. Any ideas about how can I further debug
this issue?

Thanks in advance.

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Subject Author Posted

Issue with NGINX as reverse proxy for grpc service

Víctor Enríquez August 07, 2020 10:20AM

Re: Issue with NGINX as reverse proxy for grpc service

Sergey Kandaurov August 07, 2020 12:30PM

Re: Issue with NGINX as reverse proxy for grpc service

Víctor Enríquez August 10, 2020 05:50AM



Sorry, only registered users may post in this forum.

Click here to login

Online Users

Guests: 50
Record Number of Users: 6 on February 13, 2018
Record Number of Guests: 421 on December 02, 2018
Powered by nginx      Powered by FreeBSD      PHP Powered      Powered by MariaDB      ipv6 ready