Was wondering if anyone could shed some light on this issue I am experiencing only with multiple php-fpm pool setups but not with single php-fpm pool. The issue is when a forum software like Xenforo or Invision board uses their native forum close option to turn off the forums for guests but still allow forum admins access, the forum via php issue a HTTP 503 status message. This seems to trip up and causes issues only for multiple php-fpm pool upstream setups causing alternating 503 and 502 bad gateway errors. Probably partially to do with the http_503 definition for fastcgi_next_upstream.

The upstream settings

upstream phpbackend {
zone zone_phpbackend 64k;
ip_hash;
keepalive 5;
server 127.0.0.1:9000 weight=50;
server 127.0.0.1:9002 weight=50;
server 127.0.0.1:9003 weight=50;
server 127.0.0.1:9004 weight=50;
server 127.0.0.1:9005 weight=50;
}

and relevant php-fpm changes made were to change

from single php-fpm pool

fastcgi_pass 127.0.0.1:9000;

to multiple php-fpm upstream pools

fastcgi_next_upstream error timeout http_500 http_503;
fastcgi_pass phpbackend;
fastcgi_keep_conn on;

I can replicate the issue with multiple php-fpm pool upstream setup by creating a 503.php file with contents

<?php http_response_code(503) ?>

and then refreshing the 503.php page and it will alternate between 503 and 502 errors

The access.log's alternating 503 and 502 errors excerpt

IPADDR - - [10/Nov/2016:06:07:07 +0000] "GET /503.php HTTP/1.1" 502 1672 "-" "Mozilla/5.0 snipped" "-" rt=0.000 ua="phpbackend" us="502" ut="0.000" ul="0" cs=-
IPADDR - - [10/Nov/2016:06:07:03 +0000] "GET /503.php HTTP/1.1" 503 1665 "-" "Mozilla/5.0 snipped" "-" rt=0.000 ua="127.0.0.1:9004, 127.0.0.1:9002, 127.0.0.1:9005, 127.0.0.1:9003, 127.0.0.1:9000" us="502, 502, 502, 502, 503" ut="0.000, 0.000, 0.000, 0.000, 0.000" ul="0, 0, 0, 0, 0" cs=-
IPADDR - - [10/Nov/2016:06:07:05 +0000] "GET /503.php HTTP/1.1" 502 1672 "-" "Mozilla/5.0 snipped" "-" rt=0.000 ua="phpbackend" us="502" ut="0.000" ul="0" cs=-
IPADDR - - [10/Nov/2016:06:07:07 +0000] "GET /503.php HTTP/1.1" 502 1672 "-" "Mozilla/5.0 snipped" "-" rt=0.000 ua="phpbackend" us="502" ut="0.000" ul="0" cs=-

using log format below

log_format main_ext '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for" '
'rt=$request_time ua="$upstream_addr" '
'us="$upstream_status" ut="$upstream_response_time" '
'ul="$upstream_response_length" '
'cs=$upstream_cache_status' ;

Using nginx 1.11.5 with PHP 5.6.27