Hello,

I wondered if you could help? We have a strange intermittent problem.

We use the Phoenix Server pattern[1] which means we throw away our servers and rebuild them from scratch rather than upgrade them. We do this at least once a day. We use Puppet to configure our boxes which run Ubuntu Precise. We run a number of nginx servers for different purposes (one for hosting monitoring dashboards, one for static content etc.), though all are very simple and similar.

Our Puppet config, for nginx, is very simple, we drop in a single nginx.conf (see below) and replace default-site.

What we've noticed is that sometimes nginx is found wanting and we are left with a server without nginx running. This happens about one in every hundred or less deploys so is very,very intermittent.

If we ssh onto the box and start nginx manually it works fine. We've found this happens on boxes with different site configurations (so sometimes the monitoring server, sometimes the static content).

The big problem we have is that there appears to be no log entries anywhere, in syslog or /var/log/nginx which suggest whether nginx started or even if it experienced any problems on startup - we've also found this with 'healthy' nginx installs, nginx doesn't seem to log any application messages anywhere.

On one occasion I was lucky enough to find a single access.log.1 which had a single 400 entry with a timestamp that matched the puppet run. This seems to suggest that nginx did indeed come up for a period of time but decided to go down. I have guessed that this is to do with the ordering of the puppet script which install nginx package, which in turn starts nginx service, puppet then drops config files and tells nginx to restart (using notify => Service[nginx]). I'm suspicious that nginx is sometimes failing to complete the restart (as in it stops but doesn't start up again) but for some reason the init.d is returning a good exit status.

So, two questions really: any ideas why nginx would not be starting? And how do we configure nginx so it gives some log output so we can attempt to diagnose nginx instances that fail to start up?

Many thanks
PGM
[1] http://martinfowler.com/bliki/PhoenixServer.html

Our nginx.conf:

user www-data;
worker_processes 4;
pid /var/run/nginx.pid;

events {
worker_connections 768;
}

http {
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
server_tokens off;

include /etc/nginx/mime.types;
default_type application/octet-stream;

access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;

gzip on;
gzip_disable "msie6";

include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;
}

I have discovered that nginx returns a status code 0 when the init.d script fails to start.

If I force an error I see output in /var/log/nginx/error.log so I'm not sure why we're not getting any output there. Any clues?

Below I have the output of a fresh, default installation of nginx but something is already running on port 80 causing the startup to fail. However, as you can see the status code is 0.

ubuntu@graphite-tc2cj:~$ sudo service nginx start
Starting nginx: nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
nginx: [emerg] still could not bind()
nginx.
ubuntu@graphite-tc2cj:~$ echo $?
0
ubuntu@graphite-tc2cj:~$ ps -aef | grep nginx
ubuntu 24422 23821 0 13:31 pts/0 00:00:00 grep --color=auto nginx

Make sure you stop the apache2 service and remove apache2 during the automated fresh install phase.