another attempt : # configuration file /etc/nginx/nginx.conf: #Core Functionality user nobody; worker_processes 8; pid /var/run/nginx.pid; pcre_jit on; error_log /var/log/nginx/error_log; #error_log /home/abackup/debug.log debug; worker_rlimit_nofile 300000; #Load Dynamic Modules include /etc/nginx/modules.d/*.load; events { worker_connections 8192; use epoll; multiby JohnCarne - Nginx Mailing List - English
I could not paste output of nginx -T, even truncating to 2500 lines instead of 158000 I get : Please shorten your messages, the body is too large.by JohnCarne - Nginx Mailing List - English
@] BR via Nginx : Idea coming right out of the blue: have you given a thought on compiling nginx (+ gradually modules) with valgrind? You should know pretty quickly if something is wrong. Thanks for this idea which can really improve the eng process... Brotli seems the main suspected issue Anoop just told me : **the reload is changed to USR1 for stats//// I will upgrade soon,by JohnCarne - Nginx Mailing List - English
Thanks for your feebacks. I do my best seriously to communicate what I can ! We note this issue on very busy server only, it will be hard to remove all modules on such busy server, and reconfig all, but not impossible. Anoop is on the case on smallest servers, and succeed to see the issue at small scale, he will troubleshoot with sometimes... Yes, 1 of the 3rd party module is leaking memoryby JohnCarne - Nginx Mailing List - English
Thanks for info, Anoop will read this, he is subscribedby JohnCarne - Nginx Mailing List - English
cpanel stat generation cause thet nginx makes a lot of reload to grab new file descriptor... no issue on that Issue is nginx, I show you situation now with 2.58% used for 1 work, which is same value for others, but gloably, nginx uses now 2.58%, this number is increasing slowly at the rythm of nginx reloads asked... # ps alx | grep nginx 5 99 711213 913692 20 0 3917936 3397132 ep_by JohnCarne - Nginx Mailing List - English
I let dev Anoop answer to you... he has a clue about the issue : https://github.com/SpiderLabs/ModSecurity-nginx/issues/45by JohnCarne - Nginx Mailing List - English
It looks like i don't speak english properly to be understood, others will open a thread on this issue, and may be explain betterby JohnCarne - Nginx Mailing List - English
We described it properly when opening ticket, I reformulate : Usually, 1 nginx worker process consumes 1.16-2% of RAM maximum on this server, and it remain stable. For some days after nginx upgrades, every overnight, during daily stat generation process of cpanel which happens on overnight like set, there is many nginx reloads due to stat generation (= normal), but this is now causing an everby JohnCarne - Nginx Mailing List - English
Uprgraded to last nginx version : memory still increase it seemsby JohnCarne - Nginx Mailing List - English
After upgrade and recompilation, issue is much less important, it increase only by 0.17% for an antire stats processing cycle, but issue remain unsolved.... # nginx -V nginx version: nginx/1.11.12 built by gcc 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) built with LibreSSL 2.5.1 TLS SNI support enabled configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/etcby JohnCarne - Nginx Mailing List - English
We made a lot of tests, removing brotli, geoip, we can't solve the issue i suspect : nginx: cache manager process simple bug !by JohnCarne - Nginx Mailing List - English
We only user nginx as proxy on concerned serverby JohnCarne - Nginx Mailing List - English
Hi, We suspect an issue on cpanel server from last update nginx. Every overnight, there is many nginx reloads due to stat generation process : We see an ever increasing memory use of ngixn worker process, usually it says around 1-2%, we now see it cumulating without coming back to normal each overnight, we are now after 3 days at 7% memory. Thanksby JohnCarne - Nginx Mailing List - English
Im now testing what said sys nginx : worker_processes 1; Try 1 first and the error is fixed you can increase it to 4 or 8by JohnCarne - Nginx Mailing List - English
also : 2016/09/26 10:26:22 14146#14146: posix_memalign(16, 16384) failed (12: Cannot allocate memory)by JohnCarne - Nginx Mailing List - English
just now 2016/09/26 10:18:52 5027#5027: malloc(4096) failed (12: Cannot allocate memory) 2016/09/26 10:18:53 5043#5043: malloc(4096) failed (12: Cannot allocate memory) 2016/09/26 10:18:54 5048#5048: malloc(4096) failed (12: Cannot allocate memory) 2016/09/26 10:18:54 5066#5066: malloc(4096) failed (12: Cannot allocate memory) 2016/09/26 10:18:55 5076#5076: malloc(4096) failed (12: Cannoby JohnCarne - Nginx Mailing List - English
No error after 24 hours now, nginx version without modules was 1 part of the solution Now I tweak aio nginx with this : directio_alignment 4k;by JohnCarne - Nginx Mailing List - English
I confirm we still not escaped with error which appeared just now : 2016/09/25 09:22:15 461680#461680: posix_memalign(16, 16384) failed (12: Cannot allocate memory)by JohnCarne - Nginx Mailing List - English
Maxim, After 29 hours error re-appeared jus tonce, which is much less than before I see a correlation on my monit system at this exact time : apache traffic had a peak, which equals to a big download peak I'm now thinking to nginx tweaks i have not done yet I now enlarge from 64m to client_max_body_size 256m; From 256k client_body_buffer_size 512k; add : send_timeout 300sby JohnCarne - Nginx Mailing List - English
Thank you for your feeback We reverted now to 1.11.3 with no brotly, and geoip module, (this version had caused no issue)by JohnCarne - Nginx Mailing List - English
nginx -V nginx version: nginx/1.11.4 built by gcc 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC) built with OpenSSL 1.0.2h 3 May 2016 TLS SNI support enabled configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/etc/nginx/modules --with-openssl=./openssl-1.0.2h --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error_log --http-log-path=/var/log/nginx/aby JohnCarne - Nginx Mailing List - English
sysctl tweaked at maximum already # Kernel sysctl configuration file for Red Hat Linux # # For binary values, 0 is disabled, 1 is enabled. See sysctl(8) and # sysctl.conf(5) for more details. # Tweak for nginx workers/connections added 16/09/2016 for issue investigation on pisix error in nginx logs net.core.somaxconn = 512 net.core.netdev_max_backlog = 512 net.ipv4.tcp_max_syn_backlogby JohnCarne - Nginx Mailing List - English
I add my config at server level #Core Functionality user nobody; worker_processes auto; worker_rlimit_nofile 50000; thread_pool iopool threads=32 max_queue=65536; pid /var/run/nginx.pid; error_log /var/log/nginx/error_log; #error_log /home/abackup/debug.log debug; #Load Dynamic Modules #include /etc/nginx/conf.d/dynamic_modules_custom.conf; events { worker_connections 2048;by JohnCarne - Nginx Mailing List - English
Hello, # nginx -v nginx version: nginx/1.11.4 We are now after 13 days we observer suddenly in nginx logs this in an intempestive manner, and causing nginx to reload, causing slow down on server : posix_memalign(16, 16384) failed (12: Cannot allocate memory) This happens after our upgrade to last nginx version through nDeploy. I called in nginx sysadmin, Ndeploy sysadmin too, and finaby JohnCarne - Nginx Mailing List - English