I'll bet that's it! There is nothing in my configuration that makes the IP addresses of the containers in the Docker network stay fixed. I would not be surprised if, when two or more containers are restarted (as they probably are every once in a while when logrotate runs), some or all of them may exchange IP addresses.
I will try to duplicate this so I can post the results here, but in any case I will find out how to assign specific IP addresses to the containers in the Docker configuration and do that from now on. Your observation that nginx looks up the IP once and assumes it will not change would explain what is going on.
I can't thank you enough, as this was driving me crazy.
Thank you also for your other very helpful suggestions (reloading nginx instead of restarting, forcing DNS lookups).
________________________________
> I have no idea what triggers this behavior. Once it happens, the only
> thing that can be done to correct it is to restart nginx. After that
> (could be minutes, hours, or days), the server will function as
> expected once again. Since I am using this setup in several
> production servers, at first I created a cron job to restart nginx
> every day, then every hour, and finally I decided to poll the sites
> on each server every five minutes, so that if the responses don’t
> look right I can restart nginx without having users experience a
> lengthy interruption.
Did you try reload instead of a restart? That's usually enough for
getting nginx update the sources, and is transparent to your users.
As for the actual problem, as I understand you have 4 docker
containers:
- aaa.com (Rails app)
- bbb.com (Rails app)
- ccc.com (Rails app)
- proxy (nginx, with the static assets for the 3 sites)
Do the ip addresses for the rails sites change over time?
Mind that nginx will query the hostname only once (at startup/reload),
*and use that same ip forever*
If the other containers switched ips, that would produce the exact
behavior that you are seeing.
You can force nginx to requery dns by using a variable
see https://forum.nginx.org/read.php?2,215830,215832#msg-215832
_______________________________________________
nginx mailing list
nginx@nginx.org
https://mailman.nginx.org/mailman/listinfo/nginx
