Welcome! Log In Create A New Profile

Advanced

Re: Support for 3rd party zlib libraries

Maxim Dounin
July 29, 2014 10:18PM
Hello!

On Tue, Jul 29, 2014 at 10:46:03PM +0200, Richard Stanway wrote:

> Hello,
> I recently came across a modified version of zlib with code contributed by
> Intel [1] that makes use of modern CPU instructions to increase
> performance. In testing, the performance gains seemed substantial, however
> when I tried to use this version with nginx, the following alert types
> appeared in the error_log on gzip requests:
>
> 2014/07/30 05:20:29 [alert] 22285#0: *739837 gzip filter failed to use
> preallocated memory: 65536 of 65520 while sending to client
> 2014/07/30 05:40:42 [alert] 29462#0: *230460 gzip filter failed to use
> preallocated memory: 1040 of 4112
>
> It appears nginx is pre-allocating a buffer based on the original zlib
> memory usage patterns, however the Intel optimized version has slightly
> higher memory requirements due to padding for SSE functions etc.
>
> Is there a chance this version could be supported by nginx, or a
> configuration option made available to control the allocation size?
>
> Thanks.
>
> [1] https://github.com/jtkukunas/zlib

This version indeed uses larger allocations (much larger in some
cases), and also uses different sizes which confuses nginx code
when it tries to allocate 8k for deflate state.

Quick hack to do correct preallocations for this zlib version:

--- a/src/http/modules/ngx_http_gzip_filter_module.c Wed Jul 09 12:27:15 2014 -0700
+++ b/src/http/modules/ngx_http_gzip_filter_module.c Wed Jul 30 05:57:50 2014 +0400
@@ -521,7 +521,16 @@ ngx_http_gzip_filter_memory(ngx_http_req
* *) 5920 bytes on amd64 and sparc64
*/

+#if 0
ctx->allocated = 8192 + (1 << (wbits + 2)) + (1 << (memlevel + 9));
+#endif
+
+ if (conf->level == 1) {
+ wbits = 13;
+ }
+
+ ctx->allocated = 8192 + 16 + (1 << (wbits + 2))
+ + (1 << (ngx_max(memlevel, 8) + 8)) + (1 << (memlevel + 8));
}


@@ -987,7 +996,7 @@ ngx_http_gzip_filter_alloc(void *opaque,

alloc = items * size;

- if (alloc % 512 != 0 && alloc < 8192) {
+ if (alloc % 512 != 0 && alloc < 8192 && items == 1) {

/*
* The zlib deflate_state allocation, it takes about 6K,

This is expected to work with normal zlib as well, but will be
suboptimal from memory usage point of view.

Note well that alerts logged aren't really serious - nginx is able
to handle such cases properly, and will fallback to normal pool
allocations instead of using preallocated block. It's just not
something it expects to ever happen, so it logs alerts to make
sure it's noticed. With different zlib libraries out there, we
probably want to just silence the alert, keeping preallocation
tuned for normal zlib.

--
Maxim Dounin
http://nginx.org/

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Subject Author Posted

Support for 3rd party zlib libraries

Richard Stanway July 29, 2014 04:48PM

Re: Support for 3rd party zlib libraries

Yichun Zhang (agentzh) July 29, 2014 05:24PM

Re: Support for 3rd party zlib libraries

Richard Stanway July 29, 2014 06:48PM

Re: Support for 3rd party zlib libraries

Yichun Zhang (agentzh) July 29, 2014 07:08PM

Re: Support for 3rd party zlib libraries

Piotr Sikora July 29, 2014 07:10PM

Re: Support for 3rd party zlib libraries

Yichun Zhang (agentzh) July 29, 2014 07:16PM

Re: Support for 3rd party zlib libraries

Maxim Dounin July 29, 2014 08:06PM

Re: Support for 3rd party zlib libraries

Maxim Dounin July 29, 2014 10:18PM



Sorry, only registered users may post in this forum.

Click here to login

Online Users

Guests: 71
Record Number of Users: 6 on February 13, 2018
Record Number of Guests: 421 on December 02, 2018
Powered by nginx      Powered by FreeBSD      PHP Powered      Powered by MariaDB      ipv6 ready