Yes. Firewall would be another option. But before to that, i would like to try out all options at nginx level if one or other would resolve the issue at nginx layer itself.

cant we put accept() filters? or
how the deny option works? can we use deny option to not to accept any new connections if number of connections already exceeds max limit from a client IP.?
are there any third party modules available for nginx to embed firewall functionality? something reliable !!

My objective is, using limit_conn directive, when number of connections exceeding limit, instead of sending 503, or 444, just do not accept any new connections from that specific IP only(if a client is opening 10000 connections at a time, it should be fine to not accept connections from that IP citing the reason that it could be malicious).

Thoughts !!

Thanks.