I've been writing a custom load test that performs nginx reload (with same actual config) and noticed that sometimes a single connection get's dropped during reload. The client was in Java, nginx on localhost in Docker under RHEL 7.5 and about 6k req/sec.

Can somebody comment - if this is expected behavior or possibly bug/configuration error?

I've managed to dump traffic and find that single request that failed (GET in packet 14968 fails):
https://drive.google.com/file/d/1I-orMdoZ-zCTiCBFJszWyba2qsO-c1oz/view?usp=sharing

The connection has served several requests at this point already and it goes like this:
>GET
<200
>GET
<200
>GET
<FIN

Next I've installed nginx on host (Ubuntu 17.10) directly and run wrk against it, performing single manual reload in the process - got 100 socket read errors:

$ wrk -t1 -c200 -d10s http://localhost:8080/news/index.html
Running 10s test @ http://localhost:8080/news/index.html
1 threads and 200 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 33.05ms 56.88ms 230.82ms 83.12%
Req/Sec 48.53k 4.88k 94.30k 96.00%
482892 requests in 10.00s, 770.43MB read
Socket errors: connect 0, read 100, write 0, timeout 0
Requests/sec: 48273.36
Transfer/sec: 77.02MB

I've also tried manually reproduce the case with no luck:

1. Launch nc -l as upstream
2. Send a single curl request to upstream via nginx and let it hang
3. Perform reload
4. Request is still hanging, old workers remain alive, new workers launched as expected.

During all cases - no errors were logged in error.log, only notice messages about reload.

-----------------

Nginx version in "in docker" case:

[root@89afa6c35cb4 /]# nginx -V
nginx version: nginx/1.14.1
built by gcc 4.8.5 20150623 (Red Hat 4.8.5-28) (GCC)
built with OpenSSL 1.0.2k-fips 26 Jan 2017
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib64/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -fPIC' --with-ld-opt='-Wl,-z,relro -Wl,-z,now -pie'

[root@89afa6c35cb4 /]# uname -a
Linux 89afa6c35cb4 4.13.0-46-generic #51-Ubuntu SMP Tue Jun 12 12:36:29 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

---------------------------

nginx version in "on host" case:

$ nginx -V
nginx version: nginx/1.12.1 (Ubuntu)
built with OpenSSL 1.0.2g 1 Mar 2016
TLS SNI support enabled
configure arguments: --with-cc-opt='-g -O2 -fdebug-prefix-map=/build/nginx-K5HqCe/nginx-1.12.1=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -Wdate-time -D_FORTIFY_SOURCE=2' --with-ld-opt='-Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,now -fPIC' --prefix=/usr/share/nginx --conf-path=/etc/nginx/nginx.conf --http-log-path=/var/log/nginx/access.log --error-log-path=/var/log/nginx/error.log --lock-path=/var/lock/nginx.lock --pid-path=/run/nginx.pid --modules-path=/usr/lib/nginx/modules --http-client-body-temp-path=/var/lib/nginx/body --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-proxy-temp-path=/var/lib/nginx/proxy --http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-debug --with-pcre-jit --with-http_ssl_module --with-http_stub_status_module --with-http_realip_module --with-http_auth_request_module --with-http_v2_module --with-http_dav_module --with-http_slice_module --with-threads --with-http_addition_module --with-http_geoip_module=dynamic --with-http_gunzip_module --with-http_gzip_static_module --with-http_image_filter_module=dynamic --with-http_sub_module --with-http_xslt_module=dynamic --with-stream=dynamic --with-stream_ssl_module --with-mail=dynamic --with-mail_ssl_module

$ uname -a
Linux user-MS-7850 4.13.0-46-generic #51-Ubuntu SMP Tue Jun 12 12:36:29 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

-------------------------------------------

nginx config in all cases:

worker_processes 1;

error_log /var/log/nginx/error.log info;
pid /var/run/nginx.pid;

events {
worker_connections 1024;
}

http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
client_max_body_size 2048m;

access_log /dev/null;

sendfile on;

keepalive_timeout 65;

gzip on;
gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript application/javascript;

include /etc/nginx/upstream.conf.d/*.conf;

server {
listen 8080 default;
charset utf-8;

include /etc/nginx/location.conf.d/*.conf;
}
}