Hello!
On Tue, Jul 29, 2014 at 10:46:03PM +0200, Richard Stanway wrote:
> Hello,
> I recently came across a modified version of zlib with code contributed by
> Intel [1] that makes use of modern CPU instructions to increase
> performance. In testing, the performance gains seemed substantial, however
> when I tried to use this version with nginx, the following alert types
> appeared in the error_log on gzip requests:
>
> 2014/07/30 05:20:29 [alert] 22285#0: *739837 gzip filter failed to use
> preallocated memory: 65536 of 65520 while sending to client
> 2014/07/30 05:40:42 [alert] 29462#0: *230460 gzip filter failed to use
> preallocated memory: 1040 of 4112
>
> It appears nginx is pre-allocating a buffer based on the original zlib
> memory usage patterns, however the Intel optimized version has slightly
> higher memory requirements due to padding for SSE functions etc.
>
> Is there a chance this version could be supported by nginx, or a
> configuration option made available to control the allocation size?
>
> Thanks.
>
> [1] https://github.com/jtkukunas/zlib
This version indeed uses larger allocations (much larger in some
cases), and also uses different sizes which confuses nginx code
when it tries to allocate 8k for deflate state.
Quick hack to do correct preallocations for this zlib version:
--- a/src/http/modules/ngx_http_gzip_filter_module.c Wed Jul 09 12:27:15 2014 -0700
+++ b/src/http/modules/ngx_http_gzip_filter_module.c Wed Jul 30 05:57:50 2014 +0400
@@ -521,7 +521,16 @@ ngx_http_gzip_filter_memory(ngx_http_req
* *) 5920 bytes on amd64 and sparc64
*/
+#if 0
ctx->allocated = 8192 + (1 << (wbits + 2)) + (1 << (memlevel + 9));
+#endif
+
+ if (conf->level == 1) {
+ wbits = 13;
+ }
+
+ ctx->allocated = 8192 + 16 + (1 << (wbits + 2))
+ + (1 << (ngx_max(memlevel, 8) + 8)) + (1 << (memlevel + 8));
}
@@ -987,7 +996,7 @@ ngx_http_gzip_filter_alloc(void *opaque,
alloc = items * size;
- if (alloc % 512 != 0 && alloc < 8192) {
+ if (alloc % 512 != 0 && alloc < 8192 && items == 1) {
/*
* The zlib deflate_state allocation, it takes about 6K,
This is expected to work with normal zlib as well, but will be
suboptimal from memory usage point of view.
Note well that alerts logged aren't really serious - nginx is able
to handle such cases properly, and will fallback to normal pool
allocations instead of using preallocated block. It's just not
something it expects to ever happen, so it logs alerts to make
sure it's noticed. With different zlib libraries out there, we
probably want to just silence the alert, keeping preallocation
tuned for normal zlib.
--
Maxim Dounin
http://nginx.org/
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
