Hi Maxim,
I enabled debug and the memalign call is happening on nginx reloads and the
ENOMEM happen sometimes on the reload(not on all reloads)
2018/08/02 05:59:08 [notice] 872052#872052: signal process started
2018/08/02 05:59:23 [notice] 871570#871570: signal 1 (SIGHUP) received from
872052, reconfiguring
2018/08/02 05:59:23 [debug] 871570#871570: wake up, sigio 0
2018/08/02 05:59:23 [notice] 871570#871570: reconfiguring
2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign:
0000000002B0DA00:16384 @16 === > the memalign call on reload
2018/08/02 05:59:23 [debug] 871570#871570: malloc: 00000000087924D0:4560
2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign:
000000000E442E00:16384 @16
2018/08/02 05:59:23 [debug] 871570#871570: malloc: 0000000005650850:4096
20
2018/08/02 05:48:49 [debug] 871275#871275: bind() xxxx:443 #71
2018/08/02 05:48:49 [debug] 871275#871275: bind() xxxx:443 #72
2018/08/02 05:48:49 [debug] 871275#871275: bind() xxxx:443 #73
2018/08/02 05:48:49 [debug] 871275#871275: bind() xxxx:443 #74
2018/08/02 05:48:49 [debug] 871275#871275: add cleanup: 000000005340D728
2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000024D3260:4096
2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000517BAF10:4096
2018/08/02 05:48:49 [debug] 871275#871275: malloc: 0000000053854FC0:4096
2018/08/02 05:48:49 [debug] 871275#871275: malloc: 0000000053855FD0:4096
2018/08/02 05:48:49 [debug] 871275#871275: malloc: 0000000053856FE0:4096
2018/08/02 05:48:49 [debug] 871275#871275: malloc: 0000000053857FF0:4096
2018/08/02 05:48:49 [debug] 871275#871275: posix_memalign:
0000000053859000:16384 @16
2018/08/02 05:48:49 [debug] 871275#871275: malloc: 000000005385D010:4096
2018/08/02 05:48:49 [debug] 871275#871275: malloc: 000000005385E020:4096
2018/08/02 05:48:49 [debug] 871275#871275: malloc: 000000005385F030:4096
2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536CD160:4096
2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536CE170:4096
2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536CF180:4096
2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536D0190:4096
2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536D11A0:4096
2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536D21B0:4096
2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536D31C0:4096
2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536D41D0:4096
2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536D51E0:4096
2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536D61F0:4096
2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536D7200:4096
2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536D8210:4096
2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536D9220:4096
Infact there are lot of such calls during a reload
2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign:
00000000BA17ED00:16384 @16
2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign:
00000000BA1B0FF0:16384 @16
2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign:
00000000BA1E12C0:16384 @16
2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign:
00000000BA211590:16384 @16
2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign:
00000000BA243880:16384 @16
2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign:
00000000BA271B30:16384 @16
2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign:
00000000BA2A3E20:16384 @16
2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign:
00000000BA2D20D0:16384 @16
2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign:
00000000BA3063E0:16384 @16
2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign:
00000000BA334690:16384 @16
2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign:
00000000BA366980:16384 @16
2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign:
00000000BA396C50:16384 @16
2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign:
00000000BA3C8F40:16384 @16
2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign:
00000000BA3F9210:16384 @16
2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign:
00000000BA4294E0:16384 @16
2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign:
00000000BA45B7D0:16384 @16
2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign:
00000000BA489A80:16384 @16
2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign:
00000000BA4BBD70:16384 @16
2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign:
00000000BA4EA020:16384 @16
2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign:
00000000BA51E330:16384 @16
2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign:
00000000BA54C5E0:16384 @16
2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign:
00000000BA57E8D0:16384 @16
2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign:
00000000BA5AEBA0:16384 @16
2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign:
00000000BA5DEE70:16384 @16
2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign:
00000000BA611160:16384 @16
2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign:
00000000BA641430:16384 @16
2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign:
00000000BA671700:16384 @16
2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign:
00000000BA6A29E0:16384 @16
2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign:
00000000BA6D5CE0:16384 @16
2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign:
00000000BA707FD0:16384 @16
2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign:
00000000BA736280:16384 @16
2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign:
00000000BA768570:16384 @16
2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign:
00000000BA796820:16384 @16
2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign:
00000000BA7CAB30:16384 @16
2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign:
00000000BA7F8DE0:16384 @16
2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign:
00000000BA82B0D0:16384 @16
2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign:
00000000BA85B3A0:16384 @16
What is perplexing is that the system has enough free (available RAM)
#############
# free -g
total used free shared buff/cache
available
Mem: 125 54 24 8 46
58
Swap: 0 0 0
#############
# ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 514579
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 514579
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
#########################################
There is no other thing limiting memory allocation
Any way to prevent this or probably identify/prevent this
On Tue, Jul 31, 2018 at 7:08 PM Maxim Dounin <mdounin@mdounin.ru> wrote:
> Hello!
>
> On Tue, Jul 31, 2018 at 09:52:29AM +0530, Anoop Alias wrote:
>
> > I am repeatedly seeing errors like
> >
> > ######################
> > 2018/07/31 03:46:33 [emerg] 2854560#2854560: posix_memalign(16, 16384)
> > failed (12: Cannot allocate memory)
> > 2018/07/31 03:54:09 [emerg] 2890190#2890190: posix_memalign(16, 16384)
> > failed (12: Cannot allocate memory)
> > 2018/07/31 04:08:36 [emerg] 2939230#2939230: posix_memalign(16, 16384)
> > failed (12: Cannot allocate memory)
> > 2018/07/31 04:24:48 [emerg] 2992650#2992650: posix_memalign(16, 16384)
> > failed (12: Cannot allocate memory)
> > 2018/07/31 04:42:09 [emerg] 3053092#3053092: posix_memalign(16, 16384)
> > failed (12: Cannot allocate memory)
> > 2018/07/31 04:42:17 [emerg] 3053335#3053335: posix_memalign(16, 16384)
> > failed (12: Cannot allocate memory)
> > 2018/07/31 04:42:28 [emerg] 3053937#3053937: posix_memalign(16, 16384)
> > failed (12: Cannot allocate memory)
> > 2018/07/31 04:47:54 [emerg] 3070638#3070638: posix_memalign(16, 16384)
> > failed (12: Cannot allocate memory)
> > ####################
> >
> > on a few servers
> >
> > The servers have enough memory free and the swap usage is 0, yet somehow
> > the kernel denies the posix_memalign with ENOMEM ( this is what I think
> is
> > happening!)
> >
> > The numbers requested are always 16, 16k . This makes me suspicious
> >
> > I have no setting in nginx.conf that reference a 16k
> >
> > Is there any chance of finding out what requests this and why this is not
> > fulfilled
>
> There are at least some buffers which default to 16k - for
> example, ssl_buffer_size (http://nginx.org/r/ssl_buffer_size).
>
> You may try debugging log to futher find out where the particular
> allocation happens, see here for details:
>
> http://nginx.org/en/docs/debugging_log.html
>
> But I don't really think it worth the effort. The error is pretty
> clear, and it's better to focus on why these allocations are
> denied. Likely you are hitting some limit.
>
> --
> Maxim Dounin
> http://mdounin.ru/
> _______________________________________________
> nginx mailing list
> nginx@nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx
>
--
*Anoop P Alias*
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
