Hi all,

In our production environment, we have several nginx servers running on Ubuntu that serve files from a very large (several PBs) NFS mounted storage.
Usually the storage responds pretty fast, but occasionally it can be very slow. Since we are using aio, when a file read operation runs slow, it's not that bad - the specific HTTP request will just take a longer time to complete.
However, we have seen cases in which the file open itself can take a long time to complete. Since the open is synchronous, when the open is slow all active requests on the same nginx worker process are delayed.
One way to mitigate the problem may be to increase the number of nginx workers, to some number well above the number of CPU cores. This will make each worker handle less requests and therefore less requests will be delayed due to a slow open, but this solution sounds far from ideal.
Another possibility (that requires some development) may be to create a thread that will perform the task of opening the file. The main thread will wait on the completion of the open-thread asynchronously, and will be available to handle other requests until the open completes. The child thread can either be created per request (many requests probably won't even need to open a file, thanks to the caching of open file handles), or alternatively some fancier thread pooling mechanism could be developed.

I'd love to hear any thoughts / ideas on this subject

Thanks,

Eran