Welcome! Log In Create A New Profile

Advanced

NFS Access Error while writing logs on mounted device

Posted by MPtiBot 
NFS Access Error while writing logs on mounted device
March 09, 2016 04:51AM
Good morning,

I am facing a very annoying issue with Nginx access and error logs.

For a reason I cannot mention, I absolutely need to write my nginx logs on a NFS partition mounted on the server.
For this, I mentionned the path to the error and access logs files (each one on a different nfs partition).
The file is properly created bu the Nginx master but only a few logs are written (most of them are lost).

To understand the problem, I separated temporarily the error log and I observed "permission denied" alerts:
2016/03/09 10:13:03 [alert] 1698#0: *13 write() to "<PATH to NFS LOG>/access.log" failed (13: Permission denied) while logging request, client: <server IP>, server: <server IP>, request: "GET /home HTTP/1.1", upstream: "http://<server IP>:<server Port>/home", host: "<server IP>:<server Port>"

A tcpdump gives me NFS3ERR_ACCES error packet at the same time.

You'll tell me it is user permission error to access the NFS mount, but logs are sometimes written and I only have this issue with Nginx (other tools works fine with it).

If you are aware of this, what should I do (in nfs server/client conf or patch for nginx) to fix this issue.

Thank you for your concern,
MPtiBot
Re: NFS Access Error while writing logs on mounted device
March 09, 2016 05:35AM
A possible solution would be to use a logserver (syslog) where you write your logs to which in turn writes the logs to NFS.

---
nginx for Windows http://nginx-win.ecsds.eu/
Re: NFS Access Error while writing logs on mounted device
March 09, 2016 06:22AM
Thank you for your quick answer itpp2012!

I retain your proposition, but I am currently not able to add any component to the server.
However, I found something which looks to work and might be related to NFS:
If I add the x right to the full path of the logs for all users (even if the server is executed as root). I have no error and everything is logged.
NFS would needs the x right to process the getattr request?

I noticed that if the file is already opened by another application (tail -f for example), this problem does not appear, even without the x right.
Re: NFS Access Error while writing logs on mounted device
March 09, 2016 06:32AM
No idea, we use log servers for collecting logs, writing and pushing logs to other log servers for analysis. These are tcp connections where the end point deals with writing to disk, nginx is never writing to disk by itself.

---
nginx for Windows http://nginx-win.ecsds.eu/
Re: NFS Access Error while writing logs on mounted device
March 11, 2016 07:20AM
I think I found the reason of this issue:

The unfsd service manual tells in the nfs3 bug section that all the path needs to be in exec rights for the user, and that the server executes the requests with the same uid, gid as the client (except for root if not autorised - which is not my case).
http://linux.die.net/man/8/unfsd

My client running Nginx is running in root for the master and in applicative user for the workers. The getattr requests sent to the nfs server when the file is not cached are done in applicative user (sent by the workers apparently) and the write requests are done by the master (in root).

As I had the wrong owner on one of my folders the NFS server refused my getattr request. When the file is already opened in root by another application such as tail -f, the client has no need to send this request.

Conclusion: Just verify the rights and owner of all the folders on the server!

Hope this might be useful if some of you gets the same problem :)
Sorry, only registered users may post in this forum.

Click here to login

Online Users

Guests: 294
Record Number of Users: 8 on April 13, 2023
Record Number of Guests: 421 on December 02, 2018
Powered by nginx      Powered by FreeBSD      PHP Powered      Powered by MariaDB      ipv6 ready