Welcome! Log In Create A New Profile

Advanced

Failed disk + proxy_intercept_errors

February 12, 2020 10:36AM
Hi,
In our deployment we do have configuration of proxy cache with multiple hard drives. Because of performance we don't have any RAID on these devices. That means we have to handle even a situation when drive dies, sometime.

After disk failure of proxy_cache_path device nginx usually starts serving users with http500. So I've had an idea we may use proxy_intercept_errors but I end up with inconsistent state: ~60 files are handled as expected, but after that every connection is terminated prematurely without a single byte sent. In access.log there is http 200.

I broke just ext4 FS (dd if=/dev/zero of=/dev/sdc bs=1k count=$((1024*100))) and I'm using nginx 1.17.7 on Linux

Relevant snippet from my configuration:
```
location ~ ^/mylocation/ {
set $spindle_bucket cache_01;
include "snippets/spindle_cache_locations_uspinclude";
proxy_cache DISK_$spindle_bucket;

proxy_pass $backend;
proxy_cache_key $uri;
proxy_cache_revalidate on;
proxy_cache_use_stale off;

recursive_error_pages on;
proxy_intercept_errors on;
error_page 400 401 402 403 405 406 407 408 409 410 411 412 413 414 415 416 417 418 420 422 423 424 426 428 429 431 444 449 450 451 = @myotherlocation;
error_page 500 501 502 503 504 505 506 507 508 510 511 = @myotherlocation;
}

# same as ^/mylocation/ but without proxy_intercept_errors and with a single spare drive only
location @myotherlocation {
include "snippets/spindle_cache_locations_uspinclude";
set $spindle_bucket "spare_01";
proxy_cache DISK_$spindle_bucket;

proxy_pass $backend;
proxy_cache_key $uri;
proxy_cache_revalidate on;
proxy_cache_use_stale error timeout invalid_header updating http_500 http_502 http_503 http_504;
}
```

In snippets/spindle_cache_locations_uspinclude I do have:
```
proxy_buffers 64 8k;

proxy_set_header Host our.akamaized.net;

proxy_cache_valid 200 720d;
proxy_cache_valid 206 720d;
#proxy_cache_valid 301 1d;
#proxy_cache_valid 302 10m;
#proxy_cache_valid any 1s;
proxy_http_version 1.1;
proxy_set_header Connection "";
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

```
```
$ curl -v http://myserver/mylocation/66666666-1111-1111-0000-01234567/2ab5355f-5508-438d-acfc-686469877fb3.ism/2ab5355f-5508-438d-acfc-686469877fb3-video_2=1481000-177.m4s
* Trying myipv6...
* TCP_NODELAY set
* Connected to myserver (myipv6) port 80 (#0)
> GET /mylocation/66666666-1111-1111-0000-01234567/2ab5355f-5508-438d-acfc-686469877fb3.ism/2ab5355f-55$
8-438d-acfc-686469877fb3-video_2=1481000-177.m4s HTTP/1.1
> Host: myserver
> User-Agent: curl/7.52.1
> Accept: */*
>
* Curl_http_done: called premature == 0
* Empty reply from server
* Connection #0 to host myserver left intact
curl: (52) Empty reply from server
```

Am I doing something wrong or is this a bug? Because of the inconsistency I tend to the 2nd. But I'm not sure at all :-)
Subject Author Posted

Failed disk + proxy_intercept_errors

chocholo3 February 12, 2020 10:36AM

Re: Failed disk + proxy_intercept_errors

Maxim Dounin February 13, 2020 10:00AM

Re: Failed disk + proxy_intercept_errors

chocholo3 February 14, 2020 04:14AM

Re: Failed disk + proxy_intercept_errors

Maxim Dounin February 17, 2020 10:56AM



Sorry, only registered users may post in this forum.

Click here to login

Online Users

Guests: 233
Record Number of Users: 8 on April 13, 2023
Record Number of Guests: 421 on December 02, 2018
Powered by nginx      Powered by FreeBSD      PHP Powered      Powered by MariaDB      ipv6 ready