livepeer-api: add AMQP-aware liveness probe (/healthz/amqp) to detect broken publish channels
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 88
- Forks
- 37
- PR merge metrics
- No merged PRs in 30d
Description
## Summary
We hit a production incident on 2026-05-02 where 2 of 4 `prod-livepeer-api` pods in `fra` had broken AMQP publish channels but the existing `/healthz` endpoint kept returning 200. Result: every `request-upload` and webhook publish that landed on those pods hung 60s waiting on a publisher confirm that never arrived → Cloudflare 524 → users saw "network error" and assets stuck in `uploading` phase indefinitely.
The pods recovered only after a manual `kubectl delete pod`.
## Evidence
- Stack trace from `livepeer-api` logs:
```
Error: Timeout publishing message events.stream.idle to queue
at o._channelPublish (/usr/local/src/store/queue.ts:336:19)
at o.publish (/usr/local/src/store/queue.ts:239:5)
at o.publishWebhook (/usr/local/src/store/queue.ts:226:5)
```
- Affected pods (uptime 8-12h since last restart) had **80 / 72 timeouts in 10 min**.
- Healthy pods on the same deployment had **0**.
- AMQP cluster itself was healthy: nodes 11-50% memory, no alarms, normal publish rate cluster-wide.
- Restarting the 2 affected pods immediately resolved the issue (0 timeouts after restart).
## Root cause hypothesis
After a TCP reconnect, `amqp-connection-manager` can land the recovered channel in a half-broken state where the channel reports "open" but publisher confirms never come back. The application library has no way to distinguish this from a healthy idle channel, and existing `/healthz` only checks HTTP, not the AMQP publish path.
## Proposed fix
Add `GET /healthz/amqp` that:
1. Publishes a no-op message to a dedicated probe queue (e.g. `healthz_probe_`) with publisher confirms enabled.
2. Returns 200 if the confirm comes back within 5s, 503 otherwise.
3. Counts as "alive" only when AMQP publish actually works end-to-end.
Then update the `livepeer-infra` Helm chart's `livenessProbe` to call `/healthz/amqp` with `failureThreshold: 3, periodSeconds: 30`. Kubernetes will then auto-restart pods where the AMQP channel goes bad — no human intervention.
## Workaround in place
A CronJob in `livepeer/livepeer-infra` that periodically scans pod logs for \`Timeout publishing message\` count and `kubectl delete pod` if it exceeds a threshold. This is a band-aid; the proper fix is this issue.
## References
- Incident debugging session: 2026-05-02, fra cluster, pods `prod-livepeer-api-55cf8f7b85-2xt2r` and `-dxmxn`
- Source of timeout: `packages/api/src/store/queue.ts:336` (`_channelPublish`)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with packages/api/src/store/queue.ts:336 and trace the existing publish path, then locate the current /healthz entry point. Review the livepeer-infra Helm chart's livenessProbe configuration. Done means the AMQP probe reports publisher-confirm success or failure within the stated limits and the chart uses it with the requested Kubernetes thresholds.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- helm, kubernetes, typescript
- Domain
- api, backend, devops, infrastructure
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100