oxidecomputer / oxidecomputer/omicron

disk creation saga needs different mechanism to wait on creation

Open
#996 7 comments 0 reactions 1 assignee View on GitHub

@smklein is already working on this.

Since May 2, 2022.

Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Description

In the last two Friday demos we tried a disk creation that appeared to work, but Nexus's state said "creating" for way longer than expected (minutes). The reason appeared to be exponential backoff in the saga, with each timeout triggering an attempt to create the disk again (using a request that would succeed if the disk were already created):
https://github.com/oxidecomputer/omicron/blob/fd3dab12f1cbfdd9268449319e3db5eda8257136/nexus/src/sagas.rs#L1228-L1271

I don't think it makes sense to use exponential backoff for this case because there's no indication that the remote side is overloaded -- it just isn't finished yet. The internal_service_policy() being used at L1266 is intended for background connections or requests to internal services where we want to retry indefinitely -- definitely nothing latency-sensitive. We could create one that's much more aggressive (e.g., once/second), but really, this isn't retrying a failure, it's just waiting for something to happen. There are other ways to do this:

  1. For instance creation, we have the sled send us a notification when the instance state changes. I like this pattern because it also works for the case where the state changes unexpectedly (e.g., a bhyve crash), though I think it creates a different problem if the notification is received by a different Nexus than the one operating the saga.
  2. Josh suggested a long poll -- instead of having the request complete immediately with a status meaning "it's not ready yet", it could wait a bounded amount of time (say, 30 seconds). If the request completes within that, the server responds immediately saying so. If not, it responds with "not ready yet" and the client immediately retries.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.