bug: Health check state diverges across nginx workers — unhealthy node keeps receiving traffic
- Dominant language
- Lua
- Stars
- 17.1k
- Forks
- 2.9k
- Avg merge
- 3d 16h
- Merged PRs (30d)
- 63
Description
## Description
We use a priority-based upstream (`nodes[].priority`) with active + passive health checks to fail over from a primary node to backup nodes when the primary becomes unhealthy. We're seeing the primary node continue to receive real traffic for minutes after one nginx worker process has already logged that it crossed the `unhealthy.http_failures` threshold for that node.
This reproduces consistently on **3.13.0, 3.15.0, and 3.18.0** with an identical upstream config — it does not appear to be tied to any specific APISIX version.
## Upstream config (sanitized)
```yaml
upstreams:
- id: upstream-api-service-priority-failover
type: roundrobin
scheme: https
pass_host: node
checks:
active:
type: https
timeout: 2
http_path: /internal-health
https_verify_certificate: false
healthy:
interval: 3
http_statuses: [200, 404]
successes: 2
unhealthy:
interval: 3
http_statuses: [500, 502, 503, 504]
http_failures: 2
tcp_failures: 2
timeouts: 2
passive:
type: https
healthy:
http_statuses: [200, 201]
successes: 2
unhealthy:
http_statuses: [500, 502, 503, 504]
http_failures: 2
tcp_failures: 2
timeouts: 2
nodes:
- host: primary.example.internal
port: 443
weight: 100
priority: 0
- host: fallback-a.example.internal
port: 443
weight: 50
priority: -1
- host: fallback-b.example.internal
port: 443
weight: 50
priority: -1
```
## Reproduction / evidence
Deployment runs with multiple nginx worker processes per pod, and multiple pod replicas behind a Kubernetes Service. We added `$pid` to our access log format to correlate real requests with the worker process that handled them, alongside APISIX's own `[healthcheck]` warn logs (which already include the worker's OS pid in the `#:` prefix).
```
2026-08-26T14:37:34Z [warn] 65#65: [lua] healthcheck.lua:1394: log(): [healthcheck]
(upstream#/upstreams/upstream-api-service-priority-failover)
unhealthy HTTP increment (2/2) for '10.0.1.100(10.0.1.100:443)', context: ngx.timer
```
→ worker process **65** in pod `api-gateway-abc123def-11111` has just crossed the unhealthy threshold for the primary node.
6.5 minutes later, a real request is served **by the same pod**, but by a **different worker process (pid 70)**, and is still routed to the same primary node:
```json
{
"timestamp": "2026-08-26T14:44:04Z",
"pod_name": "api-gateway-abc123def-11111",
"pid": "70",
"upstream_addr": "10.0.1.100:443",
"status": "201"
}
```
No further `[healthcheck]` log lines were emitted for this target on this pod in between — in particular, no "healthy SUCCESS" recovery line, which `lua-resty-healthcheck`'s `incr_counter()` would emit on any real state transition back to healthy (an unhealthy→healthy transition doesn't short-circuit the way same-state reports do). That suggests worker 65's checker state genuinely stayed "unhealthy" the whole time, and worker 70 simply never converged to it.
## Environment
- APISIX 3.13.0, 3.15.0, and 3.18.0 (reproduces on all three) — same upstream config on all.
- Multiple nginx worker processes per instance with auto worker config, multiple Apisix replicas.
- Priority-based upstream nodes (primary + 2 fallback), active (https) + passive checks enabled.
## Expected behavior
Once any worker process's active health checker crosses the unhealthy threshold for a node, all worker processes in that instance should stop routing traffic to that node until it's confirmed healthy again — health check state should not diverge silently between workers within the same process group.
Contributor guide
Research direction
Start with the healthcheck.lua:1394 warning and trace how active and passive checks use lua-resty-healthcheck state across nginx workers. Compare the worker-specific checker state with the request routing path under the priority-based upstream configuration. Done means an unhealthy node is not selected by other workers until a real healthy transition is observed, with the reported multi-worker reproduction covered by verification.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes, lua, nginx
- Domain
- backend, devops, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100