Hello,

We have a setup where we have requests coming into a cache server which is in front of an app server. The requests hit nginx on the cache server, which makes the request to a process running on the cache server. That process will return either the requested content (if in the cache) or a 404. In the case of a 404, nginx on the cache server should then turn around and make the same request to the app server. The app server serves the content, and then nginx on the app server should turn around and return the response to the client.

The problem we are seeing is that this works fine for requests of the form /admin/fan_pages, but when requesting css or png content (like /stylesheets/common.css , or /images/admin/fan_pages/refresh.png ), it is returning the correct content, but with a 404 code. In this case, the local process on the cache server is returning a 404, the nginx on the cache server is requesting the page from the app server (which returns the page with a 200), but then the cache server is turning around and serving the correct content to the client with a 404.

Anybody have any ideas on how/why this is happening? I'll include config files and output below. If there's a simpler or better setup to accomplish what we're trying to do, I'm happy to try that as well.

Thanks,
--Austin

ngrep output (.185 is the cache server, .165 is the app server):


#
T 24.227.153.188:54091 -> 10.102.38.185:80 [AP]
GET /stylesheets/common.css HTTP/1.1.
Host: test5.involver.com.
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2.8) Gecko/20100722 Firefox/3.6.8.
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8.
Accept-Language: en-us,en;q=0.5.
Accept-Encoding: gzip,deflate.
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7.
Keep-Alive: 115.
Connection: keep-alive.
Cookie: involver-auth=eyJlbWFpbCI6ImF1c3Rpbi5taWxsc0BpbnZvbHZlci5jb20iLCJzaWduYXR1%0AcmUiOiJLTDA5TmZvMG9QR3V2OUl3QTFkekFWOG1rU04wdkU1d2UyMzBvSjA3%0ATGhvUzJSTDNQOU5tdjRwM09FOFRcblFlUzUvWkFScmxXT29UeldIOVlONTJa%0AUEFzVzdDOXlQdUI1UUZDZURZR3dLZUgzT2RTMVpKbm96ZTY0Q1xuYjk2QUhU%0Aa25FVGdpWmRHaFA4eVJkaE13SDB3YmVkS25MejNGeC9Wd2I4WDhYSDNVM1Yw%0AMkl3RUwwRjNDXG5BS3h3ekFTMlhINHU4b0RXUVM1VkorZ2c4eTFUS2V0UCtw%0AbUVZbXVwN01pcUlXa3BDRnBBSVN2WU5TeXZcbmQ0ekhXWWhDSUE5Ulo4UXY.
.

####
T 10.102.38.185:42920 -> 10.102.38.165:80 [AP]
GET /stylesheets/common.css HTTP/1.0.
X-Real-IP: 24.227.153.188.
X-Forwarded-For: 24.227.153.188.
Host: test5.involver.com.
Connection: close.
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2.8) Gecko/20100722 Firefox/3.6.8.
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8.
Accept-Language: en-us,en;q=0.5.
Accept-Encoding: gzip,deflate.
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7.
Cookie: involver-auth=eyJlbWFpbCI6ImF1c3Rpbi5taWxsc0BpbnZvbHZlci5jb20iLCJzaWduYXR1%0AcmUiOiJLTDA5TmZvMG9QR3V2OUl3QTFkekFWOG1rU04wdkU1d2UyMzBvSjA3%0ATGhvUzJSTDNQOU5tdjRwM09FOFRcblFlUzUvWkFScmxXT29UeldIOVlONTJa%0AUEFzVzdDOXlQdUI1UUZDZURZR3dLZUgzT2RTMVpKbm96ZTY0Q1xuYjk2QUhU%0Aa25FVGdpWmRHaFA4eVJkaE13SDB3YmVkS25MejNGeC9Wd2I4WDhYSDNVM1Yw%0AMkl3RUwwRjNDXG5BS3h3ekFTMlhINHU4b0RXUVM1VkorZ2c4eTFUS2V0UCtw%0AbUVZbXVwN01pcUlXa3BDRnBBSVN2WU5TeXZcbmQ0ekhXWWhDSUE5Ulo4UXY.
.

##
T 10.102.38.165:80 -> 10.102.38.185:42920 [A]
HTTP/1.1 200 OK.
Server: nginx/0.7.65.
Date: Thu, 20 Jan 2011 01:14:02 GMT.
Content-Type: text/css.
Last-Modified: Wed, 19 Jan 2011 22:20:44 GMT.
Connection: close.
Expires: Sun, 17 Jan 2021 01:14:02 GMT.
Cache-Control: max-age=315360000.
Content-Encoding: gzip.
.
...........ZYo.H.~..E#~.n`..iIF..$.......b..@h.-.0E
$.c.........{... ............?............}~.~.....`..\.y......?.._|?Z.N.z.>...$.=....I...%.....U.....U..D.:a5_..~.nE.....e...V...V.L....rs...YQ.....d6...D.Z......Z...D.E.....y.. ..........)e.....7....]yD.tV[^.......1*J.=?#6{q&`.d.G..s...^_...2+x]-2..q.~...t.K.....4!.vbQ.u&`.1I.;.%.../...E......0I.m......o.....4..]u...!....+.....<{..$.Ws6.Av.Q<....UE.&......J)M\..?y7._..#k.m].@..H.:.P.G.j...X_Q..+.v+.....10Z..-.y.......xy}./..+.........Eo.-.<....Q..P....\...&.. ....."......$.9<.I.O.H?......?.)..Ed..Ncp.....].U.E...47..:.t......r~.....x..i....X..h...E.]......d..gh3uLo:....](...(".c.1..:y.=T...i'...3iW.E.Q(.XB.....PC..o.W..b..t..[....6\Q.io....H...D.......Q../..M
.5k.p@S.}.o<..[.h!k.$.Q4.'...wiR.I....C....2....Z....L..a.......Fs..B...hHo8....c.d....HxK9...'{...%}....h..e.z.>.................?._...^.,(.c ..X2..t.....P.Q.s).............N]..$.1.?..@....,s....Z....1X...M...pF..d.,......a...y.....l.P...R.'(h. ......Ek........N..G...O$..#...k..L.t...4=..E^1r.&..SK).....[].X..*j(.U..V.V1H..gQ..u.Nk..m.7&...L.M.....|K.H....,8..I8...J.....y.F....0..T.4.
x.,!....b.O..&.....q.9.f.u...E..<......8...8.A......B&@.&../...........].....OX.~.+nR.z,n..*.....{ ....{S......]L........`.......LoG..[3..3.W|.f...t]......}/..X..5.P...Q.w.b.P..Ix.e(.h.T.0.......y.....l[..J...a2..>.R.{%e .,.2..@....ad....[..5.../p...;..Uu.....Y....t4.D._.K..'.U....R.z.*..+...Kp.....L"......"...tb...0.uB..p$+.@D5.{..+Uk..*....M.?[4..D....S.^~x..a
..J.F+.i'...=L2._.........X...._.....2..]..k...o(.lp....\....\)...*....T.......L.O..z...Z9.......RP/I.&......(...q.7[.Pb!..7.>...<./..'.P....c"...A..c
.sf8..........d.MA.;`....H..6.ut2K..@....g7....P..6.3p..Z..."....l....}V60.x.y.#...)y.P..X...C}.t...9....7..b..)9.T.....m...^7.%..2!~L..1R....%...:uBt.3....@..B.....&.d....r:.....`.A..'..B.y.......|........F...b....n.@.../.y.O.$v.)....u*B..P......f..x8nD.6...T..z.L.{.j...|..#...b......M.u...|)td..<TA.3:.t`.R4...2>......G...B.$.wp>6@q6.8.0.P&j..].l.w..A....+....%N......h....@.x.C.%i.....:Y].._.l......./,..Q..Lj....|+)...........7.-R.....(..].C......D$..pq.c@.K.......... ..2..k....$e.P.... .v..r....U.sx...]...$$$
.4.&.........G.....-...o1...G<....[...~i.....xp.ar.....n...|....W......B+...z.+c5
....B.N...>.R...zX.....~.s..J..............0..r.6.5....?L>...........V4&...Z.,uWWWNVl.l7:.0..}Jv....:....=...H..>...2p.z..(..W.:......D..X.I'.a../H....(9.o..6[..G..9..p.kq._. ....&..$C.r...2y:!.d].....`....Y.|.D.~d..........S.K.....>}.....V.m..%...<..-wE....'
?.#1...1NZ.p.........'=.{....Ic^......4...E5.....lWf.x}.n.JT...{..iU-...y...5[.+a...2...J.....?I..//....\*e4a.i...I/9.^....NR.......i.M...i3}:...
##
T 10.102.38.165:80 -> 10.102.38.185:42920 [AP]
.?A...H....Q....?z.'.....6....L..]9......)...a_.:]B.X......K..RO.eQ......7.ko.$R.C..P.8....,.$.....@.[.E$L...q.e.%8...cY.....c#.
.K..~!'....Y.<y...{>-.v....}^...t.7.N1.t;.x.vq..(^..=..E|..../..R.;.......l<.....0/.+t.....q_.?}....J../.56.A..`...Ad..2g..........0.9.2...=..&..X......4..1....u.......QhqIQ='..@7z..0....+..
###
T 10.102.38.185:80 -> 24.227.153.188:54091 [A]
HTTP/1.1 404 Not Found.
Server: nginx/0.7.65.
Date: Thu, 20 Jan 2011 01:14:02 GMT.
Content-Type: text/css.
Transfer-Encoding: chunked.
Connection: keep-alive.
Last-Modified: Wed, 19 Jan 2011 22:20:44 GMT.
Expires: Sun, 17 Jan 2021 01:14:02 GMT.
Cache-Control: max-age=315360000.
Content-Encoding: gzip.
.
b86.
...........ZYo.H.~..E#~.n`..iIF..$.......b..@h.-.0E
$.c.........{... ............?............}~.~.....`..\.y......?.._|?Z.N.z.>...$.=....I...%.....U.....U..D.:a5_..~.nE.....e...V...V.L....rs...YQ.....d6...D.Z......Z...D.E.....y.. ..........)e.....7....]yD.tV[^.......1*J.=?#6{q&`.d.G..s...^_...2+x]-2..q.~...t.K.....4!.vbQ.u&`.1I.;.%.../...E......0I.m......o.....4..]u...!....+.....<{..$.Ws6.Av.Q<....UE.&......J)M\..?y7._..#k.m].@..H.:.P.G.j...X_Q..+.v+.....10Z..-.y.......xy}./..+.........Eo.-.<....Q..P....\...&.. ....."......$.9<.I.O.H?......?.)..Ed..Ncp.....].U.E...47..:.t......r~.....x..i....X..h...E.]......d..gh3uLo:....](...(".c.1..:y.=T...i'...3iW.E.Q(.XB.....PC..o.W..b..t..[....6\Q.io....H...D.......Q../..M
.5k.p@S.}.o<..[.h!k.$.Q4.'...wiR.I....C....2....Z....L..a.......Fs..B...hHo8....c.d....HxK9...'{...%}....h..e.z.>.................?._...^.,(.c ..X2..t.....P.Q.s).............N]..$.1.?..@....,s....Z....1X...M...pF..d.,......a...y.....l.P...R.'(h. ......Ek........N..G...O$..#...k..L.t...4=..E^1r.&..SK).....[].X..*j(.U..V.V1H..gQ..u.Nk..m.7&...L.M.....|K.H....,8..I8...J.....y.F....0..T.4.
x.,!....b.O..&.....q.9.f.u...E..<......8...8.A......B&@.&../...........].....OX.~.+nR.z,n..*.....{ ....{S......]L........`.......LoG..[3..3.W|.f...t]......}/..X..5.P...Q.w.b.P..Ix.e(.h.T.0.......y.....l[..J...a2..>.R.{%e .,.2..@....ad....[..5.../p...;..Uu.....Y....t4.D._.K..'.U....R.z.*..+...Kp.....L"......"...tb...0.uB..p$+.@D5.{..+Uk..*....M.?[4..D....S.^~x..a
..J.F+.i'...=L2._.........X...._.....2..]..k...o(.lp....\....\)...*....T.......L.O..z...Z9.......RP/I.&......(...q.7[.Pb!..7.>...<./..'.P....c"...A..c
.sf8..........d.MA.;`....H..6.ut2K..@....g7....P..6.3p..Z..."....l....}V60.x.y.#...)y.P..X...C}.t...9....7..b..)9.T.....m...^7.%..2!~L..1R....%...:uBt.3....@..B.....&.d....r:.....`.A..'..B.y.......|........F...b....n.@.../.y.O.$v.)....u*B..P......f..x8nD.6...T..z.L.{.j...|..#...b......M.u...|)td..<TA.3:.t`.R4...2>......G...B.$.wp>6@q6.8.0.P&j..].l.w..A....+....%N......h....@.x.C.%i.....:Y].._.l......./,..Q..Lj....|+)...........7.-R.....(..].C......D$..pq.c@.K.......... ..2..k....$e.P.... .v..r....U.sx...]...$$$
.4.&.........G.....-...o1...G<....[...~i.....xp.ar.....n...|....W......B+...z.+c5
....B.N...>.R...zX.....~.s..J..............0..r.6.5....?L>...........V4&...Z.,uWWWNVl.l7:.0..}Jv....:....=...H..>...2p.z..(..W.:......D..X.I'.a../H....(9.o..6[..G..9..p.kq._. ....&..$C.r...2y:!.d].....`....Y.|.D.~d..........S.K.....>}.....V
#
T 10.102.38.185:80 -> 24.227.153.188:54091 [AP]
.m..%...<..-wE....'
?.#1...1NZ.p.........'=.{....Ic^......4...E5.....lWf.x}.n.JT...{..iU-...y...5[.+a...2...J.....?I..//....\*e4a.i...I/9.^....NR.......i.M...i3}:....?A...H....Q....?z.'.....6....L..]9......)...a_.:]B.X......K..RO.eQ......7.ko.$R.C..P.8....,.$.....@.[.E$L...q.e.%8...cY.....c#.
.K..~!'....Y.<y...{>-.v....}^...t.7.N1.t;.x.vq..(^..=..E|..../..R.;.......l<.....0/.+t.....q_.?}....J../.56.A..`...Ad..2g..........0.9.2...=..&..X......4..1....u.......QhqIQ='..@7z..0....+...

####
T 10.102.38.185:80 -> 24.227.153.188:54091 [AP]
0.
.

######################################################

nginx.conf:


user involver;
worker_processes 4;
pid /var/run/nginx.pid;
worker_rlimit_nofile 10240;

events {
worker_connections 8192;
use epoll;
}

http {

include /etc/nginx/mime.types;

default_type application/octet-stream;

log_format main '$remote_addr - $remote_user [$time_local] '
'"$request" $status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';

# main access log
access_log /data/nginx/log/nginx_access.log main;

# main error log
error_log /data/nginx/log/nginx_error.log debug;

sendfile on;

tcp_nopush on;


include /etc/nginx/apps/*.conf;

}

##########################################

apps/involver-cache-server.conf (test5.involver.com is the app server):


upstream switching_backend {
server localhost:81 weight=100;
server test5.involver.com;
}

upstream rails_backend {
server test5.involver.com;
}

upstream cache_backend {
server localhost:8190;
server localhost:8191;
server localhost:8192;
server localhost:8193;
}

server {
listen 80 default;
server_name _;
root /data/apps/involver-cache-server/current/public;

# Fail fast with a 404 for server pages that rails backend will never serve
location ~* \.(php|jsp|asp|aspx|cfm|pl|py|tpl)$ { return 404; }

include /etc/nginx/common/app.conf;
include /etc/nginx/common/proxy.conf;

location = /file_uploads {
client_max_body_size 200m;
proxy_pass http://rails_backend;
break;
}

location ~* /rest/features/(.*)/content_items/(.*)/images {
proxy_pass http://rails_backend;
break;
}

location /assets/audio_uploads { return 404; }
location /assets {
proxy_pass http://rails_backend;
break;
}

location /cache {
proxy_pass http://rails_backend;
break;
}

location / {
proxy_next_upstream http_404 http_500 http_502 http_503 http_504 error timeout;
proxy_pass http://switching_backend;
}

}

# OFFLOADED SSL
server {
listen 88;
server_name _;
root /data/apps/involver-cache-server/current/public;

# Fail fast with a 404 for server pages that rails backend will never serve
location ~* \.(php|jsp|asp|aspx|cfm|pl|py|tpl)$ { return 404; }

include /etc/nginx/common/app.conf;
include /etc/nginx/common/proxy.conf;

proxy_set_header X-FORWARDED_PROTO https;

location ~* /rest/features/(.*)/content_items/(.*)/images {
proxy_pass http://rails_backend;
break;
}

location /assets/audio_uploads { return 404; }
location /assets {
proxy_pass http://rails_backend;
break;
}

location /cache {
proxy_pass http://rails_backend;
break;
}

location / {
proxy_next_upstream http_404 http_500 http_502 http_503 http_504 error timeout;
proxy_pass http://switching_backend;
}
}

# CACHING HANDLER
server {
listen 81;
server_name _;
root /data/apps/involver-cache-server/current/public;

include /etc/nginx/common/proxy.conf;

location / {
proxy_pass http://cache_backend;
}
}


#############################################

common/app.conf:

# Max size for file uploads
client_max_body_size 100M;

# This allows people to use images and css in their maintenance.html file
if ($request_filename ~* \.(css|jpg|gif|png)$) {
break;
}

# Rewrite all the requests to the maintenance.html page if it exists.
if (-f $document_root/system/maintenance.html) {
return 503;
}

error_page 404 /404.html;
error_page 500 502 504 /500.html;

error_page 503 @503;
location @503 {
rewrite ^(.*)$ /system/maintenance.html break;
}


##################################################

common/proxy.conf:

index index.html index.htm;

# needed to forward user's IP address to rails
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;

proxy_redirect off;
proxy_max_temp_file_size 0;

server_name_in_redirect off;

# set Expire header on assets: see http://developer.yahoo.com/performance/rules.html#expires
location ~ ^/(images|javascripts|stylesheets)/ {
expires 10y;
}

# serve any existing file
if (-f $request_filename) {
break;
}

# serve any standard Rails page cache file with .html extension
if (-f $request_filename.html) {
rewrite (.*) $1.html break;
}