oxidecomputer / oxidecomputer/omicron

Possible dropped deliveries when multiple webhook receivers overlap

Open
#11,235 0 comments 0 reactions 1 assignee View on GitHub

@hawkw is already working on this.

Since Sep 3, 2026.

fault-management
Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Description

Was having Claude review some of my alerting work in the console and it highlighted an issue around the liveness probe's resend=true functionality:

A success on one receiver can suppress resends on a different receiver.

The API docs for alert_receiver_probe say that with resend=true, "any alerts for which delivery to this receiver has failed will be queued for re-delivery." Omicron implements that with rx_list_resendable_events_query in nexus/db-queries/src/db/datastore/webhook_delivery.rs:205-240. The query joins alerts to deliveries and filters the outer join on rx_id = , then excludes any alert where a NOT EXISTS subquery finds another delivery of that alert that is not in the failed state and was not a probe. The subquery filters on alert_id, state, and triggered_by only. It does not filter on rx_id.

Consider alert X sent to receivers A and B. B delivered it, A failed permanently. Probing A with resend=true should requeue X for A, but the subquery finds B's delivered record and drops X from the list. Nexus reports resends_started: 0 and A never gets X. The same thing happens if B's delivery is merely pending. Effectively, one receiver's success "marks the alert as sent" for every receiver when computing the resend set. …

The fix would be adding also_delivery.rx_id = rx_id to the subquery.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.