canonical / canonical/content-cache-operator
Backend health check does not send Host header, fails against Host-based proxy ACLs
- Dominant language
- Python
- Stars
- 1
- Forks
- 1
- Avg merge
- 1d 17h
- Merged PRs (30d)
- 8
Description
## Summary
The nginx `lua-resty-upstream-healthcheck` config generated for backend health checks sends a bare, host-less HTTP/1.0 request with no `Host` header:
```
GET / HTTP/1.0\r\n\r\n
```
(see `/etc/nginx/conf.d/lua_healthchecks.conf`, generated via `nginx_manager.py`'s `init_worker_by_lua_block` / `hc.spawn_checker{...}` call).
## Impact
Any backend that sits behind a Host-header-based network ACL or transparent proxy (e.g. a squid egress proxy that only allows traffic matching specific domains, such as ProdStack's PS7 `egress.ps7.internal:3128`) will always fail health checks and be marked permanently `DOWN`, even though the same request succeeds when a `Host` header is explicitly supplied. This happens regardless of what `cache-backend-hostname` is configured via the `cache-config` relation, since that value is only used for the actual proxied request (`proxy_pass`), not for the health check request.
### Reproduction
1. Deploy `content-cache` fronting an origin that requires domain-based routing/ACL to be reached (e.g. an origin only reachable through a transparent proxy with Host-based ACLs).
2. Configure `cache-backend-hostname` appropriately for the real proxied requests (which succeed when tested manually with the correct `Host` header).
3. Observe `curl http://localhost/nginx_backends_status` reports the backend as permanently `DOWN`.
4. Observe `/var/log/nginx/error.log` entries like:
```
[error] ... healthcheck: failed to receive status line from :80: closed, context: ngx.timer
```
5. Manually confirm the same origin IP responds `200 OK` when the correct `Host` header is supplied, but the healthcheck's bare host-less request gets the connection reset/closed by the intermediate proxy.
## Suggested fix
`lua-resty-upstream-healthcheck`'s `spawn_checker` supports a `host` field. The generated `http_req` (or a `host` option) should include the configured `cache-backend-hostname` (when set) so health checks accurately reflect real backend reachability for origins behind Host-based network policies.
## Environment
Encountered while manually deploying `content-cache` + `ingress-configurator` + `aproxy` on ProdStack 7 (PS7), with `content-cache`'s origin behind PS7's mandatory egress squid proxy (which enforces a Host-based ACL for allowed destination domains).
Contributor guide
Research direction
Start in nginx_manager.py at the init_worker_by_lua_block and hc.spawn_checker configuration that generates /etc/nginx/conf.d/lua_healthchecks.conf. Trace how cache-backend-hostname is used for proxied requests and provide it to the health check when configured. Verify that nginx_backends_status reports the backend as healthy and that the health check no longer fails behind a Host-based proxy ACL.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- nginx, python
- Domain
- backend, infrastructure
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 76/100