oxidecomputer / oxidecomputer/omicron

[sled agent] Serialize Nexus notification queue

Open
#1,917 10 comments 0 reactions 1 assignee View on GitHub

@smklein is already working on this.

Since Nov 4, 2022.

Sled Agent
Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Description

There are many spots in the Sled Agent where we send notifications up to Nexus, such as:

https://github.com/oxidecomputer/omicron/blob/17ab9fdc69022ebfc8f72e85ddc4da1e658b6d1b/sled-agent/src/server.rs#L83-L123

These notification calling sites are spread around Sled Agent. For unrelated updates, this is no problem. However, for related updates, this can unfortunately result in "earlier updates" trampling over "later ones".

Example problems:

  • If we attempt to notify Nexus about a new sled - as a "Gimlet" - but later try to notify Nexus about the sled being updated to a "Scrimlet", then the ordering of the two notifications is critical. It's important that Nexus eventually sees the sled as a Scrimlet, and if a retry loop happens to mean that the "see sled as gimlet" notification arrives last, we'd end up in an inconsistent state.
  • We can try to notify Nexus about datasets before zpools, and about zpools before the sled itself. This results in a "failure + retry", which is fine, but produces some confusing log messages.

Proposal:

  • Create a more broadly-usable "notification queue", where calls to nexus may be serialized.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.