Welcome! Log In Create A New Profile

Advanced

Extremely slow file (~5MB) upload via POST

November 10, 2019 09:57AM
Hi everyone

I'm new here, and i've searched if the problem appeared before but couldn't find anything useful.

[DESCRIPTION] I've an upstream backend service behind Niginx (1.16.1, openssl-1.1.1) which allow people upload files from their browser. The files are simply stored on disk. Nothing else is performed on them.

[SYSTEM CONFIG]
. Linux 4.15.0 Ubuntu 18.04 LTS SMP x86_64
. RAM: 32GB
. CPU: 8-Cores Intel(R) Xeon(R) CPU E3-1270 v6 @ 3.80GHz
. DISK: 1TB SSD
. NETWORK CARD: 10Gbps
. System is never under load. We usually upload 10 files per hour at max.

[DATA CONFIG]
. File size is between 5MB to 20MB

[NGINX CONFIG]

We are running Nginx 1.16.1 with TLSv1.3 support (built on openssl 1.1.1).

-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8

$ cat /etc/nginx/nginx.conf
worker_processes auto;
worker_rlimit_nofile 100000;
pid /run/nginx.pid;

error_log off; #/var/log/nginx/error.log info;

events {
worker_connections 655350;
multi_accept on;
use epoll;
}

http {
include mime.types;
default_type application/octet-stream;

server_tokens off;

keepalive_timeout 3600;

access_log off; #/var/log/nginx/access.log;
sendfile on;
tcp_nopush on;
tcp_nodelay on;

types_hash_max_size 2048;

open_file_cache max=10000 inactive=10m;
open_file_cache_valid 1h;
open_file_cache_min_uses 1;
open_file_cache_errors on;
include /etc/nginx/conf.d/*.conf;

}

-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8

$ cat /etc/nginx/conf.d/uploader.conf
server {
listen 443 ssl;

server_name BACKEND_HOST_NAME:

ssl_certificate /etc/nginx/certs/bundle.pem;
ssl_certificate_key /etc/nginx/certs/key.pem;
ssl_dhparam /etc/nginx/certs/dh.pem; ## 2048-bit

ssl_protocols TLSv1.2 TLSv1.3;

ssl_prefer_server_ciphers on;
ssl_ciphers ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384;

client_max_body_size 30m;

location / {
proxy_pass http://127.0.0.1:7777;
}
}

-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8

[PROBLEM] A 5MB file takes almost 30 seconds to upload via Nginx.
When uploading it directly to the upstream backend, it takes ~400 millisec at max.

Running strace, we've got this:
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
47.96 0.041738 11 3887 1489 read
21.73 0.018909 13 1509 epoll_wait
17.95 0.015622 22 708 writev
10.62 0.009241 13 712 write
0.47 0.000407 19 21 21 connect
0.39 0.000338 17 20 close
0.21 0.000180 8 22 epoll_ctl
0.20 0.000173 8 21 socket
0.13 0.000110 110 1 accept4
0.11 0.000095 5 21 getsockopt
0.10 0.000091 4 21 recvfrom
0.07 0.000060 3 21 ioctl
0.04 0.000037 12 3 brk
0.03 0.000023 8 3 setsockopt
------ ----------- ----------- --------- --------- ----------------
100.00 0.087024 6970 1510 total

A lot of errors in "read" calls: 1489 errors. They all correspond to (thanks again to strace):

22807 read(3, "\26\3\1\2\0\1\0\1\374\3\3\304\353\3\333\314\0\36\223\244z\246\322n\375\205\360\322\35\237_\240"..., 16709) = 517
22807 read(3, 0x559de2a23f03, 16709) = -1 EAGAIN (Resource temporarily unavailable)
22807 read(3, "\24\3\3\0\1\1\26\3\3\2\0\1\0\1\374\3\3\304\353\3\333\314\0\36\223\244z\246\322n\375\205"..., 16709) = 523
22807 read(3, 0x559de2a23f03, 16709) = -1 EAGAIN (Resource temporarily unavailable)
22807 read(3, "\27\3\3\0E\271m'\306\262\26X\36J\25lC/\202_7\241\32\342XN \357\303%\264\0"..., 16709) = 74
22807 read(3, "\27\3\3\0\245\240\204\304KJ\260\207\301\232\3147\217\357I$\243\266p+*\343L\335\6v\276\323"..., 16709) = 478
22807 read(3, 0x559de2a1e9a3, 16709) = -1 EAGAIN (Resource temporarily unavailable)
22807 read(3, "\27\3\3\0\32`\324\324\237\v\266n\300x\24\277\357z\374)\365\260F\235\24\346#A%\300\376", 16709) = 31
22807 read(3, 0x559de2a1e9a3, 16709) = -1 EAGAIN (Resource temporarily unavailable)
22807 read(3, "\27\3\3\0\177\310*W\352\265\230\357\325\177\302\275\357=\246`\246^\372\214T\206\264b\352;\273z"..., 16709) = 814
22807 read(3, 0x559de2a1e9a3, 16709) = -1 EAGAIN (Resource temporarily unavailable)
22807 read(3, "\27\3\3\0Y\330\276PNY\220\245\254E\0066\2016\355\334\237Yo\2510\253\320+\26z\342\275"..., 16709) = 644
22807 read(3, 0x559de2a229e3, 16709) = -1 EAGAIN (Resource temporarily unavailable)
22807 read(3, "\27\3\3\0Z \237j\230\f\331\222\246\325\1\272Y]\252\255%\31\257L\25\10\226\267 \253\353\367"..., 16709) = 285
22807 read(3, 0x559de2a1e9a3, 16709) = -1 EAGAIN (Resource temporarily unavailable)
22807 read(3, "\27\3\3\0\212\216j6\256\370\367\310\366Hjs\275r\276>\217\216\374\377a\375\363\4\2yr\23"..., 16709) = 176
22807 read(3, 0x559de2a1e9a3, 16709) = -1 EAGAIN (Resource temporarily unavailable)
22807 read(3, "\27\3\3\0\227K2\345P\200Ls\234\10\230f\362\221\273\270V\2371X\261|\245\315\240B\177\224"..., 16709) = 1717
22807 read(3, 0x559de2a1e9a3, 16709) = -1 EAGAIN (Resource temporarily unavailable)
22807 read(3, "\27\3\3>\232\344\316\245i\375hM\362\376\frr\340\21umx&\3311\373}\35\4\3069`"..., 16709) = 4380
22807 read(3, 0x559de2a1fabf, 11651) = -1 EAGAIN (Resource temporarily unavailable)


We tried to tune our Nginx's config, but the result is always the same:
22807 read(3, 0x559de2a1fabf, 11651) = -1 EAGAIN (Resource temporarily unavailable)


Help appreciated

/F.
Subject Author Posted

Extremely slow file (~5MB) upload via POST

frank.muller November 10, 2019 09:57AM

Re: Extremely slow file (~5MB) upload via POST

Andrei November 10, 2019 06:00PM

Re: Extremely slow file (~5MB) upload via POST

Francis Daly November 11, 2019 07:14AM



Sorry, only registered users may post in this forum.

Click here to login

Online Users

Guests: 72
Record Number of Users: 6 on February 13, 2018
Record Number of Guests: 421 on December 02, 2018
Powered by nginx      Powered by FreeBSD      PHP Powered      Powered by MariaDB      ipv6 ready