On Fri, Jun 26, 2009 at 09:06:36AM -0400, meto wrote:
> There's been a massacre at 3pm for nginx procs:
> 2009/06/26 14:56:35 3039#0: worker process 3040 exited on signal 11
> 2009/06/26 14:56:35 3039#0: worker process 3043 exited on signal 11
> 2009/06/26 14:56:36 3039#0: worker process 3041 exited on signal 11
> 2009/06/26 14:56:36 3039#0: worker process 6622 exited on signal 11
> 2009/06/26 14:56:36 3039#0: worker process 3042 exited on signal 11
> 2009/06/26 14:56:36 3039#0: worker process 6625 exited on signal 11
> 2009/06/26 14:56:36 3039#0: worker process 6623 exited on signal 11
> 2009/06/26 14:56:36 3039#0: worker process 6626 exited on signal 11
> 2009/06/26 14:56:36 3039#0: worker process 6627 exited on signal 11
> 2009/06/26 14:56:37 3039#0: worker process 6624 exited on signal 11
> 2009/06/26 14:56:37 3039#0: worker process 6630 exited on signal 11
> 2009/06/26 14:56:37 3039#0: worker process 6631 exited on signal 11
> 2009/06/26 14:56:37 3039#0: worker process 6632 exited on signal 11
> 2009/06/26 14:56:39 3039#0: worker process 6633 exited on signal 11
> 2009/06/26 14:56:39 3039#0: worker process 6634 exited on signal 11
>
> That's also viewable on the graph (http://xgame.pl/nginx/), but no coredump was saved due to chmod i believe :/ So, that was a good guess with workers dieing. I'll try to provide coredumps later on.

Compile Nginx --with-cc-opt="-O0 -g3" first. It will be a bit slower
(and noticeably larger) but should be easier to debug compared to
tightly optimised release code.

Are you using any addon modules/3rd party patches?

> The second question is why were there so many of them, since i've set worker numer to 4 and affinity proc/core. maybe thets the problem?

If you're on anyhow modern Linux, try something like this:

mkdir -p /var/lib/core
chmod 1733 /var/lib/core
echo /var/lib/core/core.pid%p.sig%s.%t > /proc/sys/kernel/core_pattern

And restart Nginx with unlimited core dump size, just to be sure:

killall nginx
ulimit -c unlimited
nginx

I have never seen Nginx segfault (apart from my own hacks), so
congratulations, I guess :)

Best regards,
Grzegorz Nosek