Anoop,

I doubt this will be the solution, but may we have a look at
/proc/buddyinfo and /proc/slabinfo the moment when nginx can't allocate
memory?

On 02.08.2018 08:15, Anoop Alias wrote:
> Hi Maxim,
>
> I enabled debug and the memalign call is happening on nginx reloads
> and the ENOMEM happen sometimes on the reload(not on all reloads)
>
> 2018/08/02 05:59:08 [notice] 872052#872052: signal process started
> 2018/08/02 05:59:23 [notice] 871570#871570: signal 1 (SIGHUP) received
> from 872052, reconfiguring
> 2018/08/02 05:59:23 [debug] 871570#871570: wake up, sigio 0
> 2018/08/02 05:59:23 [notice] 871570#871570: reconfiguring
> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign:
> 0000000002B0DA00:16384 @16      === > the memalign call on reload
> 2018/08/02 05:59:23 [debug] 871570#871570: malloc: 00000000087924D0:4560
> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign:
> 000000000E442E00:16384 @16
> 2018/08/02 05:59:23 [debug] 871570#871570: malloc: 0000000005650850:4096
> 20
>
>
>
>
> 2018/08/02 05:48:49 [debug] 871275#871275: bind() xxxx:443 #71
> 2018/08/02 05:48:49 [debug] 871275#871275: bind() xxxx:443 #72
> 2018/08/02 05:48:49 [debug] 871275#871275: bind() xxxx:443 #73
> 2018/08/02 05:48:49 [debug] 871275#871275: bind() xxxx:443 #74
> 2018/08/02 05:48:49 [debug] 871275#871275: add cleanup: 000000005340D728
> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000024D3260:4096
> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000517BAF10:4096
> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 0000000053854FC0:4096
> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 0000000053855FD0:4096
> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 0000000053856FE0:4096
> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 0000000053857FF0:4096
> 2018/08/02 05:48:49 [debug] 871275#871275: posix_memalign:
> 0000000053859000:16384 @16
> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 000000005385D010:4096
> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 000000005385E020:4096
> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 000000005385F030:4096
> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536CD160:4096
> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536CE170:4096
> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536CF180:4096
> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536D0190:4096
> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536D11A0:4096
> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536D21B0:4096
> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536D31C0:4096
> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536D41D0:4096
> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536D51E0:4096
> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536D61F0:4096
> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536D7200:4096
> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536D8210:4096
> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536D9220:4096
>
>
> Infact there are lot of such calls during a reload
>
> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign:
> 00000000BA17ED00:16384 @16
> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign:
> 00000000BA1B0FF0:16384 @16
> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign:
> 00000000BA1E12C0:16384 @16
> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign:
> 00000000BA211590:16384 @16
> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign:
> 00000000BA243880:16384 @16
> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign:
> 00000000BA271B30:16384 @16
> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign:
> 00000000BA2A3E20:16384 @16
> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign:
> 00000000BA2D20D0:16384 @16
> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign:
> 00000000BA3063E0:16384 @16
> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign:
> 00000000BA334690:16384 @16
> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign:
> 00000000BA366980:16384 @16
> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign:
> 00000000BA396C50:16384 @16
> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign:
> 00000000BA3C8F40:16384 @16
> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign:
> 00000000BA3F9210:16384 @16
> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign:
> 00000000BA4294E0:16384 @16
> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign:
> 00000000BA45B7D0:16384 @16
> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign:
> 00000000BA489A80:16384 @16
> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign:
> 00000000BA4BBD70:16384 @16
> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign:
> 00000000BA4EA020:16384 @16
> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign:
> 00000000BA51E330:16384 @16
> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign:
> 00000000BA54C5E0:16384 @16
> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign:
> 00000000BA57E8D0:16384 @16
> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign:
> 00000000BA5AEBA0:16384 @16
> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign:
> 00000000BA5DEE70:16384 @16
> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign:
> 00000000BA611160:16384 @16
> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign:
> 00000000BA641430:16384 @16
> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign:
> 00000000BA671700:16384 @16
> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign:
> 00000000BA6A29E0:16384 @16
> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign:
> 00000000BA6D5CE0:16384 @16
> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign:
> 00000000BA707FD0:16384 @16
> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign:
> 00000000BA736280:16384 @16
> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign:
> 00000000BA768570:16384 @16
> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign:
> 00000000BA796820:16384 @16
> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign:
> 00000000BA7CAB30:16384 @16
> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign:
> 00000000BA7F8DE0:16384 @16
> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign:
> 00000000BA82B0D0:16384 @16
> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign:
> 00000000BA85B3A0:16384 @16
>
>
>
> What is perplexing is that the system has enough free (available RAM)
> #############
> # free -g
>               total        used        free      shared buff/cache 
>  available
> Mem:            125          54          24           8         46   
>       58
> Swap:             0           0           0
> #############
>
> # ulimit -a
> core file size          (blocks, -c) 0
> data seg size           (kbytes, -d) unlimited
> scheduling priority             (-e) 0
> file size               (blocks, -f) unlimited
> pending signals                 (-i) 514579
> max locked memory       (kbytes, -l) 64
> max memory size         (kbytes, -m) unlimited
> open files                      (-n) 1024
> pipe size            (512 bytes, -p) 8
> POSIX message queues     (bytes, -q) 819200
> real-time priority              (-r) 0
> stack size              (kbytes, -s) 8192
> cpu time               (seconds, -t) unlimited
> max user processes              (-u) 514579
> virtual memory          (kbytes, -v) unlimited
> file locks                      (-x) unlimited
>
> #########################################
>
> There is no other thing limiting memory allocation
>
> Any way to prevent this or probably identify/prevent this
>
>
> On Tue, Jul 31, 2018 at 7:08 PM Maxim Dounin <mdounin@mdounin.ru
> <mailto:mdounin@mdounin.ru>> wrote:
>
> Hello!
>
> On Tue, Jul 31, 2018 at 09:52:29AM +0530, Anoop Alias wrote:
>
> > I am repeatedly seeing errors like
> >
> > ######################
> > 2018/07/31 03:46:33 [emerg] 2854560#2854560: posix_memalign(16,
> 16384)
> > failed (12: Cannot allocate memory)
> > 2018/07/31 03:54:09 [emerg] 2890190#2890190: posix_memalign(16,
> 16384)
> > failed (12: Cannot allocate memory)
> > 2018/07/31 04:08:36 [emerg] 2939230#2939230: posix_memalign(16,
> 16384)
> > failed (12: Cannot allocate memory)
> > 2018/07/31 04:24:48 [emerg] 2992650#2992650: posix_memalign(16,
> 16384)
> > failed (12: Cannot allocate memory)
> > 2018/07/31 04:42:09 [emerg] 3053092#3053092: posix_memalign(16,
> 16384)
> > failed (12: Cannot allocate memory)
> > 2018/07/31 04:42:17 [emerg] 3053335#3053335: posix_memalign(16,
> 16384)
> > failed (12: Cannot allocate memory)
> > 2018/07/31 04:42:28 [emerg] 3053937#3053937: posix_memalign(16,
> 16384)
> > failed (12: Cannot allocate memory)
> > 2018/07/31 04:47:54 [emerg] 3070638#3070638: posix_memalign(16,
> 16384)
> > failed (12: Cannot allocate memory)
> > ####################
> >
> > on a few servers
> >
> > The servers have enough memory free and the swap usage is 0, yet
> somehow
> > the kernel denies the posix_memalign with ENOMEM ( this is what
> I think is
> > happening!)
> >
> > The numbers requested are always 16, 16k . This makes me suspicious
> >
> > I have no setting in nginx.conf that reference a 16k
> >
> > Is there any chance of finding out what requests this and why
> this is not
> > fulfilled
>
> There are at least some buffers which default to 16k - for
> example, ssl_buffer_size (http://nginx.org/r/ssl_buffer_size).
>
> You may try debugging log to futher find out where the particular
> allocation happens, see here for details:
>
> http://nginx.org/en/docs/debugging_log.html
>
> But I don't really think it worth the effort.  The error is pretty
> clear, and it's better to focus on why these allocations are
> denied.  Likely you are hitting some limit.
>
> --
> Maxim Dounin
> http://mdounin.ru/
> _______________________________________________
> nginx mailing list
> nginx@nginx.org <mailto:nginx@nginx.org>
> http://mailman.nginx.org/mailman/listinfo/nginx
>
>
>
> --
> *Anoop P Alias*
>
>
>
> _______________________________________________
> nginx mailing list
> nginx@nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx


_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx