oxidecomputer / oxidecomputer/omicron
Possible dropped deliveries when multiple webhook receivers overlap
@hawkw is already working on this.
Since Sep 3, 2026.
- Dominant language
- Rust
- Stars
- 572
- Forks
- 97
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 96
Description
Was having Claude review some of my alerting work in the console and it highlighted an issue around the liveness probe's resend=true functionality:
A success on one receiver can suppress resends on a different receiver.
The API docs for alert_receiver_probe say that with resend=true, "any alerts for which delivery to this receiver has failed will be queued for re-delivery." Omicron implements that with rx_list_resendable_events_query in nexus/db-queries/src/db/datastore/webhook_delivery.rs:205-240. The query joins alerts to deliveries and filters the outer join on rx_id = , then excludes any alert where a NOT EXISTS subquery finds another delivery of that alert that is not in the failed state and was not a probe. The subquery filters on alert_id, state, and triggered_by only. It does not filter on rx_id.
Consider alert X sent to receivers A and B. B delivered it, A failed permanently. Probing A with resend=true should requeue X for A, but the subquery finds B's delivered record and drops X from the list. Nexus reports resends_started: 0 and A never gets X. The same thing happens if B's delivery is merely pending. Effectively, one receiver's success "marks the alert as sent" for every receiver when computing the resend set. …
The fix would be adding also_delivery.rx_id = rx_id to the subquery.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.