Welcome! Log In Create A New Profile

Advanced

Buffer reuse like gzip filter module, with pre-configured number of buffers

May 31, 2022 12:57PM
Hi,

I am writing my own filter module based on the gzip filter module. My filter module would first insert a long text (200 to 1024 KB based on the situation) at the beginning of the original response and then do some other manipulations to the original response. The pre-configured number of buffers that can be allocated per request (like the gzip_buffers directive in gzip filter module) will reduce the risk of OOM.

Here is the example code:


ngx_int_t rc;
ngx_chain_t *ch = NULL;
size_t len = 64 * 4096;

ngx_buf_t *b = ngx_create_temp_buf(r->pool, len);
if (b == NULL) {
return NGX_ERROR;
}
b->recycled = 1;
b->tag = (ngx_buf_tag_t) &ngx_http_my_filter_module;

// operation to copy 64 kb data to the ngx_buf_t b;
b->last = b->pos + len;

ch = ngx_alloc_chain_link(r->pool);
if (ch == NULL) {
goto failed;
}
ch->buf = b;
ch->next = NULL;

rc = ngx_http_next_body_filter(r, ch); // rc got NGX_OK
if (rc != NGX_OK && rc != NGX_AGAIN) {
goto failed;
}

ngx_chain_update_chains(r->pool, &ctx->free, &ctx->busy, &ch,
(ngx_buf_tag_t) &ngx_http_my_filter_module);

ngx_chain_t *cl = ch;
while (cl) {
ngx_log_error(NGX_LOG_ERR, r->connection->log, 0, "buf sz: %z", ngx_buf_size(cl->buf)); // output: buf sz: 65536
cl = cl->next;
}

// reuse the buffer b if possible


Explanation:

I created a buffer b with 64k, then I copied 64k data, then I called ngx_http_next_body_filter to send the first 64k data to the client, then I want to reuse the buffer b to send the next 64k data and so on.

The question is after ngx_http_next_body_filter, ctx->free is still NULL, because ngx_buf_size(b) got 65536 which means that the buffer b cannot be recycled. My guess even if the next_body_filter returns an NGX_OK, it doesn't guarantee any of the buffers inside the ngx_chain_t can be reused.

What should I do if I want to reuse the 64k buffer?
Subject Author Posted

Buffer reuse like gzip filter module, with pre-configured number of buffers

hanzhai May 31, 2022 12:57PM

Re: Buffer reuse like gzip filter module, with pre-configured number of buffers

Maxim Dounin May 31, 2022 04:38PM

Re: Buffer reuse like gzip filter module, with pre-configured number of buffers

hanzhai June 01, 2022 08:22AM

Re: Buffer reuse like gzip filter module, with pre-configured number of buffers

hanzhai June 01, 2022 08:45AM

Re: Buffer reuse like gzip filter module, with pre-configured number of buffers

hanzhai June 01, 2022 12:07PM



Sorry, only registered users may post in this forum.

Click here to login

Online Users

Guests: 47
Record Number of Users: 6 on February 13, 2018
Record Number of Guests: 421 on December 02, 2018
Powered by nginx      Powered by FreeBSD      PHP Powered      Powered by MariaDB      ipv6 ready