Welcome! Log In Create A New Profile

Advanced

Re: Issue with NGINX as reverse proxy for grpc service

Víctor Enríquez
August 10, 2020 05:50AM
Hi Sergey,

First thanks for your reply.

What I don't really understand is, shouldn't nginx be more strict by
default with the requests that are passed to grpc backends? I have been
reading a little bit about the GRPC protocol, and it's supposed to just
use POST requests (I might be wrong about this though). Shouldn't any
other kind of request be filtered by nginx directly (at least if it is
obvious that they are malformed)?

I am going to try to reproduce this with a docker compose file and the
standard hello grpc example, to see if it's a failure on our backend
implementation or if I can reproduce it with other grpc backends.

Other than this issue, the backend is working fine if the request is
not created with curl (the clients that are supposed to use the backend
are working fine), so I guess there is some checking on the golang's
grpc implementation to check if the request is valid, and it returns
that value RST_STREAM(INTERNAL_ERROR) (but I think we don't have
control over it AFAIK, again I might be wrong).

To me ,without being an expert, it sounds like there is something
wrong, either on the nginx side or on the grpc implementation side of
our backend, but we are not returning that error code directly.
(Perhaps the golang's grpc implementation should return other error
code that doesn't make nginx believe that the upstream is dead when it
receives a malformed request).

Again thanks for your help.

On Fri, 2020-08-07 at 19:28 +0300, Sergey Kandaurov wrote:
> > On 7 Aug 2020, at 17:18, Víctor Enríquez <victor@bitonic.nl> wrote:
> >
> > Hi,
> >
> > So we have a service exposing a grpc interface under a certain
> > location
> > and we are using nginx in front of it. The config looks like the
> > following:
> >
> > upstream grpcservers {
> > server fqdn:port;
> > server fqdn:port;
> > }
> >
> > ...
> >
> > server {
> > listen port ssl http2;
> > client_max_body_size 15m;
> > server_name fqdn;
> >
> > ssl_certificate /etc/certs/server.crt;
> > ssl_certificate_key /etc/certs/server.key;
> >
> > location /my.location. {
> > grpc_set_header X-Ip-Address $remote_addr;
> > grpc_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
> > grpc_ssl_certificate /etc/ssl/mtls-client.crt;
> > grpc_ssl_certificate_key /etc/ssl/mtls-client.key;
> > grpc_pass grpcs://grpcservers;
> > ...
> > }
> >
> > # Error responses
> > include conf.d/errors.grpc_conf; # gRPC-compliant error responses
> > default_type application/grpc; # Ensure gRPC for all error
> > responses
> >
> > } //End of the server directive
> >
> > Now we just realized that each time we do a GET / to that specific
> > port
> > under that specific location using curl --http2, the request is
> > forwarded to the backend in such a way that it makes nginx believe
> > that
> > the backend has crashed, allowing anyone to DDoS this particular
> > service by just repeteadly sending GET / request to the endpoint.
> >
> > I am seeing the following messages in the logs:
> >
> > 020/08/07 13:02:37 [error] 1100#1100: *199 upstream rejected
> > request
> > with error 2 while reading response header from upstream, client:
> > X.X.X.X, server: fqdn1, request: "POST
> > /my.location.magic.API/GetMagic
> > HTTP/2.0", upstream: "grpcs://Z.Z.Z.Z:PORT", host: "fqdn1:PORT"
>
> "error 2" means that backend responded with
> RST_STREAM(INTERNAL_ERROR),
> that is, effectively rejected processing request.
> You may want to consult with backend error log to find out the
> reason.
>

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Subject Author Posted

Issue with NGINX as reverse proxy for grpc service

Víctor Enríquez August 07, 2020 10:20AM

Re: Issue with NGINX as reverse proxy for grpc service

Sergey Kandaurov August 07, 2020 12:30PM

Re: Issue with NGINX as reverse proxy for grpc service

Víctor Enríquez August 10, 2020 05:50AM



Sorry, only registered users may post in this forum.

Click here to login

Online Users

Guests: 55
Record Number of Users: 6 on February 13, 2018
Record Number of Guests: 421 on December 02, 2018
Powered by nginx      Powered by FreeBSD      PHP Powered      Powered by MariaDB      ipv6 ready