Hello.
I have a single-digit fleet of Ubuntu servers, all running a similar configuration:
* Ubuntu 20.04LTS, current kernel via `apt`
* Nginx 1.23.1 from source, with 3rd party modules
* PHP 8.0 or 8.1 from source
One of the servers has recently had issues with what appears to be an Nginx segfault, which manifests as high CPU load, repeated `apport` processes spawning, and errors written to `apport.log`. The other servers have a similar config, but are not affected by this issue.
The process that spawns is:
/usr/bin/python3 /usr/share/apport/apport -p58000 -s11 -c0 -d1 -P58000 -u33 -g33 -- !usr!sbin!nginx
…where the `p` and `P` value corresponds to the PID. An excerpt from `apport,log` follows:
ERROR: apport (pid 58552) Sat Aug 13 07:23:24 2022: called for pid 58551, signal 11, core limit 0, dump mode 1
ERROR: apport (pid 58552) Sat Aug 13 07:23:24 2022: executable: /usr/sbin/nginx (command line "nginx:\ master\ process\ /usr/sbin/nginx\ -g\ daemon\ on;\ master_process\ on;")
ERROR: apport (pid 58552) Sat Aug 13 07:23:24 2022: is_closing_session(): no DBUS_SESSION_BUS_ADDRESS in environment
ERROR: apport (pid 58552) Sat Aug 13 07:23:24 2022: apport: report /var/crash/_usr_sbin_nginx.33.crash already exists and unseen, skipping to avoid disk usage DoS
ERROR: apport (pid 58554) Sat Aug 13 07:23:24 2022: called for pid 58553, signal 11, core limit 0, dump mode 1
ERROR: apport (pid 58554) Sat Aug 13 07:23:24 2022: executable: /usr/sbin/nginx (command line "nginx:\ master\ process\ /usr/sbin/nginx\ -g\ daemon\ on;\ master_process\ on;")
ERROR: apport (pid 58554) Sat Aug 13 07:23:24 2022: is_closing_session(): no DBUS_SESSION_BUS_ADDRESS in environment
ERROR: apport (pid 58554) Sat Aug 13 07:23:24 2022: apport: report /var/crash/_usr_sbin_nginx.33.crash already exists and unseen, skipping to avoid disk usage DoS
ERROR: apport (pid 58556) Sat Aug 13 07:23:24 2022: called for pid 58555, signal 11, core limit 0, dump mode 1
ERROR: apport (pid 58556) Sat Aug 13 07:23:24 2022: executable: /usr/sbin/nginx (command line "nginx:\ master\ process\ /usr/sbin/nginx\ -g\ daemon\ on;\ master_process\ on;")
ERROR: apport (pid 58556) Sat Aug 13 07:23:24 2022: is_closing_session(): no DBUS_SESSION_BUS_ADDRESS in environment
ERROR: apport (pid 58556) Sat Aug 13 07:23:24 2022: apport: report /var/crash/_usr_sbin_nginx.33.crash already exists and unseen, skipping to avoid disk usage DoS
ERROR: apport (pid 58560) Sat Aug 13 07:23:25 2022: called for pid 58559, signal 11, core limit 0, dump mode 1
ERROR: apport (pid 58560) Sat Aug 13 07:23:25 2022: executable: /usr/sbin/nginx (command line "nginx:\ master\ process\ /usr/sbin/nginx\ -g\ daemon\ on;\ master_process\ on;")
ERROR: apport (pid 58560) Sat Aug 13 07:23:25 2022: is_closing_session(): no DBUS_SESSION_BUS_ADDRESS in environment
ERROR: apport (pid 58560) Sat Aug 13 07:23:25 2022: apport: report /var/crash/_usr_sbin_nginx.33.crash already exists and unseen, skipping to avoid disk usage DoS
ERROR: apport (pid 58562) Sat Aug 13 07:23:25 2022: called for pid 58561, signal 11, core limit 0, dump mode 1
ERROR: apport (pid 58562) Sat Aug 13 07:23:25 2022: executable: /usr/sbin/nginx (command line "nginx:\ master\ process\ /usr/sbin/nginx\ -g\ daemon\ on;\ master_process\ on;")
ERROR: apport (pid 58562) Sat Aug 13 07:23:25 2022: is_closing_session(): no DBUS_SESSION_BUS_ADDRESS in environment
ERROR: apport (pid 58562) Sat Aug 13 07:23:25 2022: apport: report /var/crash/_usr_sbin_nginx.33.crash already exists and unseen, skipping to avoid disk usage DoS
ERROR: apport (pid 58564) Sat Aug 13 07:23:25 2022: called for pid 58563, signal 11, core limit 0, dump mode 1
ERROR: apport (pid 58564) Sat Aug 13 07:23:25 2022: executable: /usr/sbin/nginx (command line "nginx:\ master\ process\ /usr/sbin/nginx\ -g\ daemon\ on;\ master_process\ on;")
ERROR: apport (pid 58564) Sat Aug 13 07:23:25 2022: is_closing_session(): no DBUS_SESSION_BUS_ADDRESS in environment
ERROR: apport (pid 58564) Sat Aug 13 07:23:25 2022: apport: report /var/crash/_usr_sbin_nginx.33.crash already exists and unseen, skipping to avoid disk usage DoS
The processes are spawning in quick succession and are enough to take it offline as far as Nginx is concerned, though `sshd` is responsive enough for me to log in.
My gut feeling indicates a server-specific issue, perhaps hardware-related, but since this is new territory for me and my research hasn't located much apart from "this might be hardware", I wanted to check in here before I migrate the sites to a new server.
I am grateful for any guidance or advice for what might be worth looking at on the server as-is.
Thank you, and best wishes to you.