i have an upstream clause with two servers. one is the primary and the other is the backup.
upstream Service {
server 127.0.0.1:8106;
server 127.0.0.1:8100 backup;
}
periodically, the primary server marks itself "down", reloads nginx, restarts, removes the "down" status, reloads nginx, and continues.
i observe that sometimes traffic continues to go to the backup server even after the primary service has restarted.
a excerpt of the log for server processes, where the respective pid apprears in "[...]" looks like this where 1676218 took over from 1430500 while the 686680 server was the backup.
2024-09-14T11:04:55.632496+02:00 [warning] spocq[1430500]: [T] graph-store-response: service quality Service 6913A03E-7278-11EF-AFDA-DDBA947010DB
2024-09-14T11:05:02.580638+02:00 [warning] spocq[1430500]: [T] graph-store-response: service quality Service 6D37D5B8-7278-11EF-AFDA-DDBA947010DB
2024-09-14T11:05:09.546325+02:00 [warning] spocq[1430500]: [T] graph-store-response: service quality Service 715EB60C-7278-11EF-AFDA-DDBA947010DB
2024-09-14T11:05:16.581053+02:00 [warning] spocq[686680]: [T] graph-store-response: service quality Service 759001FE-7278-11EF-AFDA-DDBA947010DB
2024-09-14T11:05:24.585299+02:00 [warning] spocq[1676218]: [T] graph-store-response: service quality Service 79BD606E-7278-11EF-AFDA-DDBA947010DB
2024-09-14T11:05:31.578072+02:00 [warning] spocq[1676218]: [T] graph-store-response: service quality Service 7E7B4602-7278-11EF-AFDA-DDBA947010DB
2024-09-14T11:05:38.436851+02:00 [warning] spocq[1676218]: [T] graph-store-response: service quality Service 8294A440-7278-11EF-AFDA-DDBA947010DB
sometimes, however, the requests continue to appear at the backup server even when the primary server is active.
in a case where the 1430500 server had started at 2024-09-14T23:34:01, entries for the backup server continue to appear hours later, which indicates that nginx continues to target it with requests.
2024-09-14T00:00:07.185312+02:00 [warning] spocq[1430500]: [T] graph-store-response: service quality Service 899C7068-721B-11EF-AFD9-DDBA947010DB
2024-09-14T00:00:20.950918+02:00 [warning] spocq[1430500]: [T] graph-store-response: service quality Service 91E9FD26-721B-11EF-AFD9-DDBA947010DB
2024-09-14T01:03:58.710002+02:00 [warning] spocq[1430500]: [T] graph-store-response: service quality Service 75307B84-7224-11EF-AFD9-DDBA947010DB
2024-09-14T01:06:00.078388+02:00 [warning] spocq[686680]: [T] graph-store-response: service quality Service BDD12802-7224-11EF-AFD9-DDBA947010DB
2024-09-14T02:00:02.177556+02:00 [warning] spocq[1430500]: [T] graph-store-response: service quality Service 49FD7806-722C-11EF-AFD9-DDBA947010DB
2024-09-14T04:02:02.273035+02:00 [warning] spocq[1430500]: [T] graph-store-response: service quality Service 55484D92-723D-11EF-AFD9-DDBA947010DB
...
2024-09-14T04:42:02.125885+02:00 [warning] spocq[1430500]: [T] graph-store-response: service quality Service EBBF2CC8-7242-11EF-AFD9-DDBA947010DB
2024-09-14T08:00:37.724415+02:00 [warning] spocq[1430500]: [T] graph-store-response: service quality Service AA09B2B4-725E-11EF-AFD9-DDBA947010DB
2024-09-14T08:02:39.136554+02:00 [warning] spocq[686680]: [T] graph-store-response: service quality Service F26A7854-725E-11EF-AFD9-DDBA947010DB
2024-09-14T08:40:07.768077+02:00 [warning] spocq[1430500]: [T] graph-store-response: service quality Service 2EB166CE-7264-11EF-AFD9-DDBA947010DB
2024-09-14T08:41:20.670565+02:00 [warning] spocq[1430500]: [T] graph-store-response: service quality Service 5A2578F4-7264-11EF-AFD9-DDBA947010DB
2024-09-14T08:41:39.515107+02:00 [warning] spocq[1430500]: [T] graph-store-response: service quality Service 65642760-7264-11EF-AFD9-DDBA947010DB
is this a known issue?
is there a remedy?
