One last follow-up, to let you know what I know about this mystery.

I can reproduce it consistently and it looks like it is probably a
mongrel problem, not nginx at all. For any particular mongrel
instance, if the first request after it has been started is a POST,
then the mongrel/rails combo can not seem to find the post data. If
the mongrel instance is hit with a GET first (or a failing POST), then
all subsequent POSTs work just fine.

Using ngrep, I've verified that nginx is sending the post data to the
mongrel instances even for that first POST, so it's looking unlikely
that it has anything to do with nginx.

Thanks again for your time and help, and I'm off to google "mongrel
missing post data on first request."

Aaron


On Sun, Apr 5, 2009 at 8:28 PM, Aaron Starr <astarr@wiredquote.com> wrote:
> Maxim Dounin and Dave Cheney,
>
> Thank you for your help. I've acquired lsof and ngrep and set the
> logging level to debug for nginx, and now -- of course -- it seems to
> be working perfectly. Aaaagh!
>
> As soon as it quits sending POST data, I'll collect all the
> information you suggested, and if I can't find anything untoward, I'll
> post it here.
>
> Thanks, again, sincerely.
>
> Aaron
>
>
> On Sun, Apr 5, 2009 at 3:56 AM, Maxim Dounin <mdounin@mdounin.ru> wrote:
>> Hello!
>>
>> On Sat, Apr 04, 2009 at 03:55:31PM -0700, Aaron Starr wrote:
>>
>>> Maxim Dounin,
>>>
>>> Thanks for your response!
>>>
>>> I have several G on the logs volume, so for that reason it's probably
>>> not a disk space issue.
>>>
>>> Also, though, the log file thing seems to have been a red herring. I
>>> just went and tried the POST, again, having made no changes at all
>>> since last time, and it's not working, again. And the log files are
>>> small.
>>>
>>> So, it seems that nginx randomly starts stripping POST data when
>>> proxying to mongrel. I can't be the first person who's seen something
>>> like this! I really don't think I've done anything unusual.
>>>
>>> Can anyone think of a reason why POST data might be stripped when proxying?
>>
>> First of all, try looking into error_log.  If it doesn't help -
>> obtain debug log and post it here together with config.
>>
>> Maxim Dounin
>>
>>
>>>
>>> Thanks for any ideas!
>>>
>>> Aaron
>>>
>>>
>>>
>>> On Sat, Apr 4, 2009 at 7:38 AM, Maxim Dounin <mdounin@mdounin.ru> wrote:
>>> > Hello!
>>> >
>>> > On Sat, Apr 04, 2009 at 12:49:13AM -0700, Aaron Starr wrote:
>>> >
>>> >> Update on the missing POST data when proxying to mongrel:
>>> >>
>>> >> This seems crazy, but it seems to me that the problem is related to
>>> >> having very large, un-rotated nginx log files. I noticed that the
>>> >> files were getting too large, and removed them (and got logrotate set
>>> >> up), and now I get my POST data, again.
>>> >>
>>> >> If anyone could confirm or deny this apparently nonsensical
>>> >> explanation, that would be appreciated.
>>> >
>>> > Probably you have no space left for files in client_body_temp_path
>>> > - and as soon as you removed large old logs there are space and
>>> > everything is working again.
>>> >
>>> > Guess you have nothing in error_log due to the very same reason -
>>> > no space left on device.
>>> >
>>> > Maxim Dounin
>>> >
>>> >>
>>> >> Aaron
>>> >>
>>> >>
>>> >> On Fri, Apr 3, 2009 at 2:22 PM, Aaron Starr <astarr@wiredquote.com> wrote:
>>> >> > Hi, all. Sorry in advance for what is hopefully a simple question.
>>> >> >
>>> >> > I'm using nginx as a reverse-proxy for a few mongrel instances. Using
>>> >> > AJAX (XMLHttpRequest), I POST a small bit of JSON to the server. The
>>> >> > request is properly proxied to mongrel, but with the POST data
>>> >> > missing!
>>> >> >
>>> >> > [..]
>>> >>
>>> >
>>> >
>>>
>>
>>
>