Good day. I'm working on http://github.com/slact/nginx_http_push_module
.. and I've got this one nasty bug that I'm having a bit of trouble with.

An aside first, though. The way this thing works is by setting up an
rbtree in a shared memory zone. Waiting listener requests are queued up
in channels (represented by nodes on the shared rbtree). Naturally, when
a listener request is finished or aborted, I want it dequeued from a
channel's "waiting list". I am doing this with a request pool cleanup
handler, using the following piece of code to set it up:

//test to see if the connection was closed or something.
r->read_event_handler = ngx_http_test_reading; //r is a long-polling
listener request.
//attach a cleaner to remove the request from the channel, if need be
(if the connection goes dead or something)
ngx_http_push_listener_cleanup_t *clndata;
ngx_pool_cleanup_t *cln = ngx_pool_cleanup_add(r->pool,
sizeof(*clndata));
if (cln == NULL) { //make sure we can.
return NGX_ERROR;
}
cln->handler = (ngx_pool_cleanup_pt) ngx_http_push_listener_cleanup;
clndata = (ngx_http_push_listener_cleanup_t *) cln->data;
clndata->channel=channel;
clndata->listener=listener;
ngx_shmtx_lock(&shpool->mutex);
listener->cleanup = clndata;
ngx_shmtx_unlock(&shpool->mutex);

Trouble is, I'm getting (occasional) segfaults from accessing said
ling-polling listener requests that have already been freed. Meaning, I
suspect, the request pool cleanup handler was not called.

Also, I am unable to reproduce this when worker_processes = 1;

Some possibly relevant valgrind output is attached. Coredumps are
getting me nowhere, as said request memory will have already been freed
by the time a segfault occurs.

My questions, then:
Are there any circumstances under which a request pool cleanup handler
will fail to fire? Or am I debugging this in a completely wrong direction?
Is there a better way to have nginx call a handler upon request
termination?

Thanque,
- Leo
$valgrind --trace-children=yes --read-var-info=yes --free-fill=F0 --malloc-fill=BA --undef-value-errors=no ./nginx -p /home/leop/verk/nginx-testing/
==26926== Memcheck, a memory error detector
==26926== Copyright (C) 2002-2009, and GNU GPL'd, by Julian Seward et al.
==26926== Using Valgrind-3.5.0-Debian and LibVEX; rerun with -h for copyright info
==26926== Command: ./nginx -p /home/leop/verk/nginx-testing/
==26926==
==26926==
==26926== HEAP SUMMARY:
==26926== in use at exit: 82,086 bytes in 2,118 blocks
==26926== total heap usage: 2,649 allocs, 531 frees, 146,430 bytes allocated
==26926==
==26926== LEAK SUMMARY:
==26926== definitely lost: 0 bytes in 0 blocks
==26926== indirectly lost: 0 bytes in 0 blocks
==26926== possibly lost: 49,332 bytes in 5 blocks
==26926== still reachable: 32,754 bytes in 2,113 blocks
==26926== suppressed: 0 bytes in 0 blocks
==26926== Rerun with --leak-check=full to see details of leaked memory
==26926==
==26926== For counts of detected and suppressed errors, rerun with: -v
==26926== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

-- intermission --

$==26926== Invalid write of size 4
==26926== at 0x80CA6BC: ngx_http_push_dequeue_listener_locked (ngx_http_push_module.c:48)
==26926== by 0x80CBB68: ngx_http_push_sender_body_handler (ngx_http_push_module.c:424)
==26926== by 0x808FFF4: ngx_http_read_client_request_body (ngx_http_request_body.c:152)
==26926== by 0x80CC1A7: ngx_http_push_sender_handler (ngx_http_push_module.c:559)
==26926== by 0x807C95C: ngx_http_core_content_phase (ngx_http_core_module.c:1249)
==26926== by 0x807B8B5: ngx_http_core_run_phases (ngx_http_core_module.c:788)
==26926== by 0x807B850: ngx_http_handler (ngx_http_core_module.c:771)
==26926== by 0x8086E3A: ngx_http_process_request (ngx_http_request.c:1557)
==26926== by 0x8085ACA: ngx_http_process_request_headers (ngx_http_request.c:1019)
==26926== by 0x808537A: ngx_http_process_request_line (ngx_http_request.c:829)
==26926== by 0x8084827: ngx_http_init_request (ngx_http_request.c:505)
==26926== by 0x80682B4: ngx_event_process_posted (ngx_event_posted.c:39)
==26926== Address 0x45aa220 is 152 bytes inside a block of size 1,024 free'd
==26926== at 0x4024866: free (vg_replace_malloc.c:325)
==26926== by 0x804E855: ngx_pfree (ngx_palloc.c:288)
==26926== by 0x8088609: ngx_http_set_keepalive (ngx_http_request.c:2346)
==26926== by 0x8087ACA: ngx_http_finalize_request (ngx_http_request.c:1998)
==26926== by 0x80CBB1D: ngx_http_push_sender_body_handler (ngx_http_push_module.c:433)
==26926== by 0x808FFF4: ngx_http_read_client_request_body (ngx_http_request_body.c:152)
==26926== by 0x80CC1A7: ngx_http_push_sender_handler (ngx_http_push_module.c:559)
==26926== by 0x807C95C: ngx_http_core_content_phase (ngx_http_core_module.c:1249)
==26926== by 0x807B8B5: ngx_http_core_run_phases (ngx_http_core_module.c:788)
==26926== by 0x807B850: ngx_http_handler (ngx_http_core_module.c:771)
==26926== by 0x8086E3A: ngx_http_process_request (ngx_http_request.c:1557)
==26926== by 0x8085ACA: ngx_http_process_request_headers (ngx_http_request.c:1019)
==26926==
==26926== Invalid write of size 4
==26926== at 0x80CC221: ngx_http_push_set_listener_header (ngx_http_push_module.c:573)
==26926== by 0x80CBAB4: ngx_http_push_sender_body_handler (ngx_http_push_module.c:430)
==26926== by 0x808FFF4: ngx_http_read_client_request_body (ngx_http_request_body.c:152)
==26926== by 0x80CC1A7: ngx_http_push_sender_handler (ngx_http_push_module.c:559)
==26926== by 0x807C95C: ngx_http_core_content_phase (ngx_http_core_module.c:1249)
==26926== by 0x807B8B5: ngx_http_core_run_phases (ngx_http_core_module.c:788)
==26926== by 0x807B850: ngx_http_handler (ngx_http_core_module.c:771)
==26926== by 0x8086E3A: ngx_http_process_request (ngx_http_request.c:1557)
==26926== by 0x8085ACA: ngx_http_process_request_headers (ngx_http_request.c:1019)
==26926== by 0x808537A: ngx_http_process_request_line (ngx_http_request.c:829)
==26926== by 0x8084827: ngx_http_init_request (ngx_http_request.c:505)
==26926== by 0x80682B4: ngx_event_process_posted (ngx_event_posted.c:39)
==26926== Address 0x45a96dc is 2,124 bytes inside a block of size 4,096 free'd
==26926== at 0x4024866: free (vg_replace_malloc.c:325)
==26926== by 0x804E3EC: ngx_destroy_pool (ngx_palloc.c:86)
==26926== by 0x8089437: ngx_http_request_done (ngx_http_request.c:2821)
==26926== by 0x8088417: ngx_http_set_keepalive (ngx_http_request.c:2300)
==26926== by 0x8087ACA: ngx_http_finalize_request (ngx_http_request.c:1998)
==26926== by 0x80CBB1D: ngx_http_push_sender_body_handler (ngx_http_push_module.c:433)
==26926== by 0x808FFF4: ngx_http_read_client_request_body (ngx_http_request_body.c:152)
==26926== by 0x80CC1A7: ngx_http_push_sender_handler (ngx_http_push_module.c:559)
==26926== by 0x807C95C: ngx_http_core_content_phase (ngx_http_core_module.c:1249)
==26926== by 0x807B8B5: ngx_http_core_run_phases (ngx_http_core_module.c:788)
==26926== by 0x807B850: ngx_http_handler (ngx_http_core_module.c:771)
==26926== by 0x8086E3A: ngx_http_process_request (ngx_http_request.c:1557)
==26926==
==26926== Invalid read of size 4
==26926== at 0x80CC230: ngx_http_push_set_listener_header (ngx_http_push_module.c:574)
==26926== by 0x80CBAB4: ngx_http_push_sender_body_handler (ngx_http_push_module.c:430)
==26926== by 0x808FFF4: ngx_http_read_client_request_body (ngx_http_request_body.c:152)
==26926== by 0x80CC1A7: ngx_http_push_sender_handler (ngx_http_push_module.c:559)
==26926== by 0x807C95C: ngx_http_core_content_phase (ngx_http_core_module.c:1249)
==26926== by 0x807B8B5: ngx_http_core_run_phases (ngx_http_core_module.c:788)
==26926== by 0x807B850: ngx_http_handler (ngx_http_core_module.c:771)
==26926== by 0x8086E3A: ngx_http_process_request (ngx_http_request.c:1557)
==26926== by 0x8085ACA: ngx_http_process_request_headers (ngx_http_request.c:1019)
==26926== by 0x808537A: ngx_http_process_request_line (ngx_http_request.c:829)
==26926== by 0x8084827: ngx_http_init_request (ngx_http_request.c:505)
==26926== by 0x80682B4: ngx_event_process_posted (ngx_event_posted.c:39)
==26926== Address 0x45a95d4 is 1,860 bytes inside a block of size 4,096 free'd
==26926== at 0x4024866: free (vg_replace_malloc.c:325)
==26926== by 0x804E3EC: ngx_destroy_pool (ngx_palloc.c:86)
==26926== by 0x8089437: ngx_http_request_done (ngx_http_request.c:2821)
==26926== by 0x8088417: ngx_http_set_keepalive (ngx_http_request.c:2300)
==26926== by 0x8087ACA: ngx_http_finalize_request (ngx_http_request.c:1998)
==26926== by 0x80CBB1D: ngx_http_push_sender_body_handler (ngx_http_push_module.c:433)
==26926== by 0x808FFF4: ngx_http_read_client_request_body (ngx_http_request_body.c:152)
==26926== by 0x80CC1A7: ngx_http_push_sender_handler (ngx_http_push_module.c:559)
==26926== by 0x807C95C: ngx_http_core_content_phase (ngx_http_core_module.c:1249)
==26926== by 0x807B8B5: ngx_http_core_run_phases (ngx_http_core_module.c:788)
==26926== by 0x807B850: ngx_http_handler (ngx_http_core_module.c:771)
==26926== by 0x8086E3A: ngx_http_process_request (ngx_http_request.c:1557)
==26926==
==26926== Invalid read of size 4
==26926== at 0x804E474: ngx_palloc (ngx_palloc.c:121)
==26926== by 0x80CC23E: ngx_http_push_set_listener_header (ngx_http_push_module.c:574)
==26926== by 0x80CBAB4: ngx_http_push_sender_body_handler (ngx_http_push_module.c:430)
==26926== by 0x808FFF4: ngx_http_read_client_request_body (ngx_http_request_body.c:152)
==26926== by 0x80CC1A7: ngx_http_push_sender_handler (ngx_http_push_module.c:559)
==26926== by 0x807C95C: ngx_http_core_content_phase (ngx_http_core_module.c:1249)
==26926== by 0x807B8B5: ngx_http_core_run_phases (ngx_http_core_module.c:788)
==26926== by 0x807B850: ngx_http_handler (ngx_http_core_module.c:771)
==26926== by 0x8086E3A: ngx_http_process_request (ngx_http_request.c:1557)
==26926== by 0x8085ACA: ngx_http_process_request_headers (ngx_http_request.c:1019)
==26926== by 0x808537A: ngx_http_process_request_line (ngx_http_request.c:829)
==26926== by 0x8084827: ngx_http_init_request (ngx_http_request.c:505)
==26926== Address 0xf0f0f100 is not stack'd, malloc'd or (recently) free'd
==26926==
==26926==
==26926== Process terminating with default action of signal 11 (SIGSEGV): dumping core
==26926== Access not within mapped region at address 0xF0F0F100
==26926== at 0x804E474: ngx_palloc (ngx_palloc.c:121)
==26926== by 0x80CC23E: ngx_http_push_set_listener_header (ngx_http_push_module.c:574)
==26926== by 0x80CBAB4: ngx_http_push_sender_body_handler (ngx_http_push_module.c:430)
==26926== by 0x808FFF4: ngx_http_read_client_request_body (ngx_http_request_body.c:152)
==26926== by 0x80CC1A7: ngx_http_push_sender_handler (ngx_http_push_module.c:559)
==26926== by 0x807C95C: ngx_http_core_content_phase (ngx_http_core_module.c:1249)
==26926== by 0x807B8B5: ngx_http_core_run_phases (ngx_http_core_module.c:788)
==26926== by 0x807B850: ngx_http_handler (ngx_http_core_module.c:771)
==26926== by 0x8086E3A: ngx_http_process_request (ngx_http_request.c:1557)
==26926== by 0x8085ACA: ngx_http_process_request_headers (ngx_http_request.c:1019)
==26926== by 0x808537A: ngx_http_process_request_line (ngx_http_request.c:829)
==26926== by 0x8084827: ngx_http_init_request (ngx_http_request.c:505)
==26926== If you believe this happened as a result of a stack
==26926== overflow in your program's main thread (unlikely but
==26926== possible), you can try to increase the size of the
==26926== main thread stack using the --main-stacksize= flag.
==26926== The main thread stack size used in this run was 8388608.
==26926==
==26926== HEAP SUMMARY:
==26926== in use at exit: 739,830 bytes in 2,828 blocks
==26926== total heap usage: 4,778 allocs, 1,950 frees, 3,540,006 bytes allocated
==26926==
==26926== LEAK SUMMARY:
==26926== definitely lost: 0 bytes in 0 blocks
==26926== indirectly lost: 0 bytes in 0 blocks
==26926== possibly lost: 16,436 bytes in 2 blocks
==26926== still reachable: 723,394 bytes in 2,826 blocks
==26926== suppressed: 0 bytes in 0 blocks
==26926== Rerun with --leak-check=full to see details of leaked memory
==26926==
==26926== For counts of detected and suppressed errors, rerun with: -v
==26926== ERROR SUMMARY: 4 errors from 4 contexts (suppressed: 0 from 0)