We've found the problem and sent email to "igor@sysoev.ru", but got no answers.
It's caused in ngx_http_upstream_process_header. If r->cache->header_start is bigger than u->conf->buffer_size, u->buffer.last will be bigger than u->buffer.end, and the response header will be written to invalid memory for this buffer. But the written memory may be valid for this process, so core dump usually happens some time later, and backstrace shows somewhere else.
a patch based on nginx-1.0.4:
diff -ruNp nginx-1.0.4/src/http/ngx_http_upstream.c nginx-1.0.4_zls/src/http/ngx_http_upstream.c
--- nginx-1.0.4/src/http/ngx_http_upstream.c 2011-06-01 16:02:34.000000000 +0800
+++ nginx-1.0.4_zls/src/http/ngx_http_upstream.c 2011-06-22 00:41:32.282201935 +0800
@@ -661,6 +661,16 @@ ngx_http_upstream_cache(ngx_http_request
ngx_http_file_cache_create_key(r);
+ if (r->cache->header_start >= u->conf->buffer_size){
+ ngx_log_error(NGX_LOG_ERR, r->connection->log, 0,
+ "cache disabled because of too big cache_key (%uz vs %uz)",
+ r->cache->header_start, u->conf->buffer_size
+ );
+
+ r->cache = NULL;
+ return NGX_DECLINED;
+ }
+
switch (ngx_http_test_predicates(r, u->conf->cache_bypass)) {
case NGX_ERROR: