nginx, echo module & lua

Re: nginx, echo module & lua

Pierre — Thu, 03 May 2012 11:40:00 -0400

On 03/mai - 21:03, agentzh wrote:
> On Thu, May 3, 2012 at 6:43 PM, Pierre wrote:
> >
> >
> > using echo_subrequest_async, I directly reach :
> > 2012/05/03 12:08:25 [error] 13204#0: *1 subrequests cycle while processing "/trucalacon",
> >
>
> Which version of nginx are you using?
tested on openresty 10.0.10 abd 1.0.11
>
> >
> > Last interrogation:
> > What is the best way to discard body (I mean free the memory by body passed to subrequests ) ?
> >
>
> There's no way to release all the memory of a subrequest instantly
> because nginx subrequests share the same memory pool as their parent
> request. So for long-running requests that issue a lot of serial
> subrequests, there will be a good chance of temporary memory leaks in
> the lifetime of the main request.
ok, thx
>
> It's worth mentioning that the ngx_lua cosocket API does not suffer
> from this issue because Lua GC handles the resource lifetime there.
:)
>
> Best regards,
> -agentzh
>
> _______________________________________________
> nginx mailing list
> nginx@nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx

Re: nginx, echo module & lua

agentzh — Thu, 03 May 2012 09:04:01 -0400

On Thu, May 3, 2012 at 6:43 PM, Pierre wrote:
>
>
> using echo_subrequest_async, I directly reach :
> 2012/05/03 12:08:25 [error] 13204#0: *1 subrequests cycle while processing "/trucalacon",
>

Which version of nginx are you using?

>
> Last interrogation:
> What is the best way to discard body (I mean free the memory by body passed to subrequests ) ?
>

There's no way to release all the memory of a subrequest instantly
because nginx subrequests share the same memory pool as their parent
request. So for long-running requests that issue a lot of serial
subrequests, there will be a good chance of temporary memory leaks in
the lifetime of the main request.

It's worth mentioning that the ngx_lua cosocket API does not suffer
from this issue because Lua GC handles the resource lifetime there.

Best regards,
-agentzh

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx

Re: nginx, echo module & lua

agentzh — Thu, 03 May 2012 08:54:01 -0400

On Thu, May 3, 2012 at 3:53 PM, Antonio P.P. Almeida wrote:
> A little thread hijacking. Are there plans to provide an HTTP response
> parser? I suppose that for performance reasons that would have to
> be in C along the lines of the Redis parser.
>
> https://github.com/agentzh/lua-redis-parser
>
> That's what's missing, IMHO, in order to completely replace
> luasocket.
>

Liseen Wan has been working on the lua-resty-http library:

https://github.com/liseen/lua-resty-http

We'll extend that to support more and more HTTP features in the near future.

But I think it will remain pure Lua because introducing C may not
really help here given the performance of LuaJIT 2.0 (and it has
already been proven in lua-resty-mysql where rewriting the MySQL
packet parser in C actually slowed things down because going across
language boundary is more expensive).

But anyway we can provide pure C API for ngx_lua cosocket, just like
ngx_http_upstream, so a pure C implementation for those lua-resty-*
library will have a chance to achieve better performance than the pure
Lua implementations, but I'm not 100% sure ;)

Regards,
-agentzh

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx

Re: nginx, echo module & lua

agentzh — Thu, 03 May 2012 08:36:01 -0400

On Thu, May 3, 2012 at 7:34 AM, Pierre wrote:
>> There's already several Lua drivers based on that, including
>> memcached, redis, mysql, http, and mongodb. For instance, the
>> lua-resty-redis library:
>>
>> https://github.com/agentzh/lua-resty-redis
>>
>
> how can I combine cosocket & foreign upstream ?
> also, I didn't find any example involving cosocket between locations, or maybe
> I'm getting it wrong.
>

The cosocket API is used to talk to remote servers in TCP or unix
domain sockets directly. It does not support the subrequest model at
all so it is not used for reusing existing upstream modules.

The "cosocket" term is short for "coroutine-based socket", it is a
parallel implementation as ngx_http_usptream, see the following graph:

http://agentzh.org/misc/slides/libdrizzle-lua-nginx/#57

>> > 2. when in config: set_by_lua_file $varname luafile.lua , and luafile.lua
>> > references to ngx.ctx,
>> > I get the following error:
>> > (lua-error)...no request ctx found
>> > according to the doc, ngx.ctx is available in set_lua*, but does not work.
>> > Would anybody suggest
>> > me a workaround ?
>> >
>>
>> This is indeed a bug. Thanks for reporting this!
>>
>> I've just committed a patch for it:
>>
>> https://github.com/chaoslawful/lua-nginx-module/commit/bee1cab
>
> works like a charm, thanks !
>

Sorry, my previous patch introduced a bad memory leak and I've just
committed the following patch to fix this:

https://github.com/chaoslawful/lua-nginx-module/commit/8d2878

Best regards,
-agentzh

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx

Re: nginx, echo module & lua

Pierre — Thu, 03 May 2012 06:44:00 -0400

Investigating a bit more, here is the conclusion:

location = /store {
internal;
#test 1
#echo_subrequest_async PUT /trucalacon -q 'filename=$arg_filename' -b $echo_request_body;
#test 2
#content_by_lua '
# local filename = ngx.req.get_uri_args()["md5"]
# local res = ngx.location.capture("/trucalacon",
# { args = {filename = filename } ,
# ctx = ngx.ctx })
#';
}
location = /trucalacon {
proxy_set_header X-Real-IP $remote_addr;
proxy_pass http://otherserver;
}

using echo_subrequest_async, I directly reach :
2012/05/03 12:08:25 [error] 13204#0: *1 subrequests cycle while processing "/trucalacon",

using echo_subrequest, I get the same behaviour as ngx.location.capture

using content_by_lua + location.capture, everything is smooth.

/store is called by another lua script, which does
location.capture.multi('/store'), 20 by 20

Last interrogation:
What is the best way to discard body (I mean free the memory by body passed to subrequests ) ?

On 03/mai - 01:34, Pierre wrote:
> On 02/mai - 19:46, agentzh wrote:
> > Hello!
> >
> > On Wed, May 2, 2012 at 4:11 PM, Pierre wrote:
> > > I'm trying to use both echo module & lua, but I'm having some issues:
> > >
> > > 1. I reach quickly "subrequests cycle". Looking @ the code, subrequests is
> > > unsigned:8,
> > > but I would need thousands of subrequests sometimes. putting subrequests to
> > > unsigned just
> > > results to crashes.
> > >
> >
> > There's no way to support thousands of subrequests in a single main
> > request without patching the nginx core.
> >
> > But you're encouraged to try out ngx_lua's cosocket API to avoid
> > subrequests altogether. And the cosocket stuff does not have such
> > limitations.
> >
> > There's already several Lua drivers based on that, including
> > memcached, redis, mysql, http, and mongodb. For instance, the
> > lua-resty-redis library:
> >
> > https://github.com/agentzh/lua-resty-redis
> >
>
> how can I combine cosocket & foreign upstream ?
> also, I didn't find any example involving cosocket between locations, or maybe
> I'm getting it wrong.
>
> > > 2. when in config: set_by_lua_file $varname luafile.lua , and luafile.lua
> > > references to ngx.ctx,
> > > I get the following error:
> > > (lua-error)...no request ctx found
> > > according to the doc, ngx.ctx is available in set_lua*, but does not work.
> > > Would anybody suggest
> > > me a workaround ?
> > >
> >
> > This is indeed a bug. Thanks for reporting this!
> >
> > I've just committed a patch for it:
> >
> > https://github.com/chaoslawful/lua-nginx-module/commit/bee1cab
>
> works like a charm, thanks !
>
> >
> > Could you try out the git master HEAD on your side?
> >
> > Thanks!
> > -agentzh
> >
> > _______________________________________________
> > nginx mailing list
> > nginx@nginx.org
> > http://mailman.nginx.org/mailman/listinfo/nginx
>
> _______________________________________________
> nginx mailing list
> nginx@nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx

Re: nginx, echo module & lua

Antonio P.P. Almeida — Thu, 03 May 2012 03:54:01 -0400

A little thread hijacking. Are there plans to provide an HTTP response
parser? I suppose that for performance reasons that would have to
be in C along the lines of the Redis parser.

https://github.com/agentzh/lua-redis-parser

That's what's missing, IMHO, in order to completely replace
luasocket.

Great work as usual.

--appa

> On 02/mai - 19:46, agentzh wrote:
>> Hello!
>>
>> On Wed, May 2, 2012 at 4:11 PM, Pierre wrote:
>> > I'm trying to use both echo module & lua, but I'm having some issues:
>> >
>> > 1. I reach quickly "subrequests cycle". Looking @ the code,
>> subrequests is
>> > unsigned:8,
>> > but I would need thousands of subrequests sometimes. putting
>> subrequests to
>> > unsigned just
>> > results to crashes.
>> >
>>
>> There's no way to support thousands of subrequests in a single main
>> request without patching the nginx core.
>>
>> But you're encouraged to try out ngx_lua's cosocket API to avoid
>> subrequests altogether. And the cosocket stuff does not have such
>> limitations.
>>
>> There's already several Lua drivers based on that, including
>> memcached, redis, mysql, http, and mongodb. For instance, the
>> lua-resty-redis library:
>>
>> https://github.com/agentzh/lua-resty-redis
>>
>
> how can I combine cosocket & foreign upstream ?
> also, I didn't find any example involving cosocket between locations, or
> maybe
> I'm getting it wrong.
>
>> > 2. when in config: set_by_lua_file $varname luafile.lua , and
>> luafile.lua
>> > references to ngx.ctx,
>> > I get the following error:
>> > (lua-error)...no request ctx found
>> > according to the doc, ngx.ctx is available in set_lua*, but does not
>> work.
>> > Would anybody suggest
>> > me a workaround ?
>> >
>>
>> This is indeed a bug. Thanks for reporting this!
>>
>> I've just committed a patch for it:
>>
>> https://github.com/chaoslawful/lua-nginx-module/commit/bee1cab
>
> works like a charm, thanks !
>
>>
>> Could you try out the git master HEAD on your side?
>>
>> Thanks!
>> -agentzh
>>
>> _______________________________________________
>> nginx mailing list
>> nginx@nginx.org
>> http://mailman.nginx.org/mailman/listinfo/nginx
>
> _______________________________________________
> nginx mailing list
> nginx@nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx
>

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx

Re: nginx, echo module & lua

Pierre — Wed, 02 May 2012 19:36:00 -0400

On 02/mai - 19:46, agentzh wrote:
> Hello!
>
> On Wed, May 2, 2012 at 4:11 PM, Pierre wrote:
> > I'm trying to use both echo module & lua, but I'm having some issues:
> >
> > 1. I reach quickly "subrequests cycle". Looking @ the code, subrequests is
> > unsigned:8,
> > but I would need thousands of subrequests sometimes. putting subrequests to
> > unsigned just
> > results to crashes.
> >
>
> There's no way to support thousands of subrequests in a single main
> request without patching the nginx core.
>
> But you're encouraged to try out ngx_lua's cosocket API to avoid
> subrequests altogether. And the cosocket stuff does not have such
> limitations.
>
> There's already several Lua drivers based on that, including
> memcached, redis, mysql, http, and mongodb. For instance, the
> lua-resty-redis library:
>
> https://github.com/agentzh/lua-resty-redis
>

how can I combine cosocket & foreign upstream ?
also, I didn't find any example involving cosocket between locations, or maybe
I'm getting it wrong.

> > 2. when in config: set_by_lua_file $varname luafile.lua , and luafile.lua
> > references to ngx.ctx,
> > I get the following error:
> > (lua-error)...no request ctx found
> > according to the doc, ngx.ctx is available in set_lua*, but does not work.
> > Would anybody suggest
> > me a workaround ?
> >
>
> This is indeed a bug. Thanks for reporting this!
>
> I've just committed a patch for it:
>
> https://github.com/chaoslawful/lua-nginx-module/commit/bee1cab

works like a charm, thanks !

>
> Could you try out the git master HEAD on your side?
>
> Thanks!
> -agentzh
>
> _______________________________________________
> nginx mailing list
> nginx@nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx

Re: nginx, echo module & lua

agentzh — Wed, 02 May 2012 11:12:00 -0400

On Wed, May 2, 2012 at 9:39 PM, Pierre wrote:
>> But you're encouraged to try out ngx_lua's cosocket API to avoid
>> subrequests altogether. And the cosocket stuff does not have such
>> limitations.
> hmm, any more documentation about that ?

See the official documentation for ngx_lua cosocket:

http://wiki.nginx.org/HttpLuaModule#ngx.socket.tcp

> More about: how to implement it in my lua scripts ?
>
> is that it ? :
>
> local tcp = ngx.socket.tcp
>
> function new(self)
> return setmetatable({ sock = tcp() }, mt)
> end
> ?

See the lua-resty-redis, lua-resty-memcached, and lua-resty-mysql
libraries as examples:

https://github.com/agentzh/lua-resty-redis
https://github.com/agentzh/lua-resty-memcached
https://github.com/agentzh/lua-resty-mysql

bigplum has been working on lua-resty-mongol:

https://github.com/bigplum/lua-resty-mongol

I'm not sure about its state, but you can read the document or contact
the author directly.

Best regards,
-agentzh

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx

Re: nginx, echo module & lua

Pierre — Wed, 02 May 2012 09:40:00 -0400

On 02/mai - 19:46, agentzh wrote:
> Hello!
>
> On Wed, May 2, 2012 at 4:11 PM, Pierre wrote:
> > I'm trying to use both echo module & lua, but I'm having some issues:
> >
> > 1. I reach quickly "subrequests cycle". Looking @ the code, subrequests is
> > unsigned:8,
> > but I would need thousands of subrequests sometimes. putting subrequests to
> > unsigned just
> > results to crashes.
> >
>
> There's no way to support thousands of subrequests in a single main
> request without patching the nginx core.
ok
>
> But you're encouraged to try out ngx_lua's cosocket API to avoid
> subrequests altogether. And the cosocket stuff does not have such
> limitations.
hmm, any more documentation about that ?
More about: how to implement it in my lua scripts ?

is that it ? :

local tcp = ngx.socket.tcp

function new(self)
return setmetatable({ sock = tcp() }, mt)
end

?

>
> There's already several Lua drivers based on that, including
> memcached, redis, mysql, http, and mongodb. For instance, the
> lua-resty-redis library:
mongodb: you mean mongofs or mongodb ?
>
> https://github.com/agentzh/lua-resty-redis
>
> > 2. when in config: set_by_lua_file $varname luafile.lua , and luafile.lua
> > references to ngx.ctx,
> > I get the following error:
> > (lua-error)...no request ctx found
> > according to the doc, ngx.ctx is available in set_lua*, but does not work.
> > Would anybody suggest
> > me a workaround ?
> >
>
> This is indeed a bug. Thanks for reporting this!
>
> I've just committed a patch for it:
>
> https://github.com/chaoslawful/lua-nginx-module/commit/bee1cab
woh, thx !
>
> Could you try out the git master HEAD on your side?
I will
>
> Thanks!
> -agentzh
>
> _______________________________________________
> nginx mailing list
> nginx@nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx

Re: nginx, echo module & lua

agentzh — Wed, 02 May 2012 07:48:00 -0400

Hello!

On Wed, May 2, 2012 at 4:11 PM, Pierre wrote:
> I'm trying to use both echo module & lua, but I'm having some issues:
>
> 1. I reach quickly "subrequests cycle". Looking @ the code, subrequests is
> unsigned:8,
> but I would need thousands of subrequests sometimes. putting subrequests to
> unsigned just
> results to crashes.
>

There's no way to support thousands of subrequests in a single main
request without patching the nginx core.

But you're encouraged to try out ngx_lua's cosocket API to avoid
subrequests altogether. And the cosocket stuff does not have such
limitations.

There's already several Lua drivers based on that, including
memcached, redis, mysql, http, and mongodb. For instance, the
lua-resty-redis library:

https://github.com/agentzh/lua-resty-redis

> 2. when in config: set_by_lua_file $varname luafile.lua , and luafile.lua
> references to ngx.ctx,
> I get the following error:
> (lua-error)...no request ctx found
> according to the doc, ngx.ctx is available in set_lua*, but does not work.
> Would anybody suggest
> me a workaround ?
>

This is indeed a bug. Thanks for reporting this!

I've just committed a patch for it:

https://github.com/chaoslawful/lua-nginx-module/commit/bee1cab

Could you try out the git master HEAD on your side?

Thanks!
-agentzh

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx

nginx, echo module & lua

Pierre — Wed, 02 May 2012 04:12:00 -0400

Hi all,

I'm trying to use both echo module & lua, but I'm having some issues:

1. I reach quickly "subrequests cycle". Looking @ the code, subrequests
is unsigned:8,
but I would need thousands of subrequests sometimes. putting
subrequests to unsigned just
results to crashes.

2. when in config: set_by_lua_file $varname luafile.lua , and
luafile.lua references to ngx.ctx,
I get the following error:
(lua-error)...no request ctx found
according to the doc, ngx.ctx is available in set_lua*, but does not
work. Would anybody suggest
me a workaround ?

I'm using luajit, openresty last version.

Thanks a lot in advance,
Pierre

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx