Hi everyone
I'm new here, and i've searched if the problem appeared before but couldn't find anything useful.
[DESCRIPTION] I've an upstream backend service behind Niginx (1.16.1, openssl-1.1.1) which allow people upload files from their browser. The files are simply stored on disk. Nothing else is performed on them.
[SYSTEM CONFIG]
. Linux 4.15.0 Ubuntu 18.04 LTS SMP x86_64
. RAM: 32GB
. CPU: 8-Cores Intel(R) Xeon(R) CPU E3-1270 v6 @ 3.80GHz
. DISK: 1TB SSD
. NETWORK CARD: 10Gbps
. System is never under load. We usually upload 10 files per hour at max.
[DATA CONFIG]
. File size is between 5MB to 20MB
[NGINX CONFIG]
We are running Nginx 1.16.1 with TLSv1.3 support (built on openssl 1.1.1).
-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8
$ cat /etc/nginx/nginx.conf
worker_processes auto;
worker_rlimit_nofile 100000;
pid /run/nginx.pid;
error_log off; #/var/log/nginx/error.log info;
events {
worker_connections 655350;
multi_accept on;
use epoll;
}
http {
include mime.types;
default_type application/octet-stream;
server_tokens off;
keepalive_timeout 3600;
access_log off; #/var/log/nginx/access.log;
sendfile on;
tcp_nopush on;
tcp_nodelay on;
types_hash_max_size 2048;
open_file_cache max=10000 inactive=10m;
open_file_cache_valid 1h;
open_file_cache_min_uses 1;
open_file_cache_errors on;
include /etc/nginx/conf.d/*.conf;
}
-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8
$ cat /etc/nginx/conf.d/uploader.conf
server {
listen 443 ssl;
server_name BACKEND_HOST_NAME:
ssl_certificate /etc/nginx/certs/bundle.pem;
ssl_certificate_key /etc/nginx/certs/key.pem;
ssl_dhparam /etc/nginx/certs/dh.pem; ## 2048-bit
ssl_protocols TLSv1.2 TLSv1.3;
ssl_prefer_server_ciphers on;
ssl_ciphers ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384;
client_max_body_size 30m;
location / {
proxy_pass http://127.0.0.1:7777;
}
}
-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8
[PROBLEM] A 5MB file takes almost 30 seconds to upload via Nginx.
When uploading it directly to the upstream backend, it takes ~400 millisec at max.
Running strace, we've got this:
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
47.96 0.041738 11 3887 1489 read
21.73 0.018909 13 1509 epoll_wait
17.95 0.015622 22 708 writev
10.62 0.009241 13 712 write
0.47 0.000407 19 21 21 connect
0.39 0.000338 17 20 close
0.21 0.000180 8 22 epoll_ctl
0.20 0.000173 8 21 socket
0.13 0.000110 110 1 accept4
0.11 0.000095 5 21 getsockopt
0.10 0.000091 4 21 recvfrom
0.07 0.000060 3 21 ioctl
0.04 0.000037 12 3 brk
0.03 0.000023 8 3 setsockopt
------ ----------- ----------- --------- --------- ----------------
100.00 0.087024 6970 1510 total
A lot of errors in "read" calls: 1489 errors. They all correspond to (thanks again to strace):
22807 read(3, "\26\3\1\2\0\1\0\1\374\3\3\304\353\3\333\314\0\36\223\244z\246\322n\375\205\360\322\35\237_\240"..., 16709) = 517
22807 read(3, 0x559de2a23f03, 16709) = -1 EAGAIN (Resource temporarily unavailable)
22807 read(3, "\24\3\3\0\1\1\26\3\3\2\0\1\0\1\374\3\3\304\353\3\333\314\0\36\223\244z\246\322n\375\205"..., 16709) = 523
22807 read(3, 0x559de2a23f03, 16709) = -1 EAGAIN (Resource temporarily unavailable)
22807 read(3, "\27\3\3\0E\271m'\306\262\26X\36J\25lC/\202_7\241\32\342XN \357\303%\264\0"..., 16709) = 74
22807 read(3, "\27\3\3\0\245\240\204\304KJ\260\207\301\232\3147\217\357I$\243\266p+*\343L\335\6v\276\323"..., 16709) = 478
22807 read(3, 0x559de2a1e9a3, 16709) = -1 EAGAIN (Resource temporarily unavailable)
22807 read(3, "\27\3\3\0\32`\324\324\237\v\266n\300x\24\277\357z\374)\365\260F\235\24\346#A%\300\376", 16709) = 31
22807 read(3, 0x559de2a1e9a3, 16709) = -1 EAGAIN (Resource temporarily unavailable)
22807 read(3, "\27\3\3\0\177\310*W\352\265\230\357\325\177\302\275\357=\246`\246^\372\214T\206\264b\352;\273z"..., 16709) = 814
22807 read(3, 0x559de2a1e9a3, 16709) = -1 EAGAIN (Resource temporarily unavailable)
22807 read(3, "\27\3\3\0Y\330\276PNY\220\245\254E\0066\2016\355\334\237Yo\2510\253\320+\26z\342\275"..., 16709) = 644
22807 read(3, 0x559de2a229e3, 16709) = -1 EAGAIN (Resource temporarily unavailable)
22807 read(3, "\27\3\3\0Z \237j\230\f\331\222\246\325\1\272Y]\252\255%\31\257L\25\10\226\267 \253\353\367"..., 16709) = 285
22807 read(3, 0x559de2a1e9a3, 16709) = -1 EAGAIN (Resource temporarily unavailable)
22807 read(3, "\27\3\3\0\212\216j6\256\370\367\310\366Hjs\275r\276>\217\216\374\377a\375\363\4\2yr\23"..., 16709) = 176
22807 read(3, 0x559de2a1e9a3, 16709) = -1 EAGAIN (Resource temporarily unavailable)
22807 read(3, "\27\3\3\0\227K2\345P\200Ls\234\10\230f\362\221\273\270V\2371X\261|\245\315\240B\177\224"..., 16709) = 1717
22807 read(3, 0x559de2a1e9a3, 16709) = -1 EAGAIN (Resource temporarily unavailable)
22807 read(3, "\27\3\3>\232\344\316\245i\375hM\362\376\frr\340\21umx&\3311\373}\35\4\3069`"..., 16709) = 4380
22807 read(3, 0x559de2a1fabf, 11651) = -1 EAGAIN (Resource temporarily unavailable)
We tried to tune our Nginx's config, but the result is always the same:
22807 read(3, 0x559de2a1fabf, 11651) = -1 EAGAIN (Resource temporarily unavailable)
Help appreciated
/F.
