oxidecomputer / oxidecomputer/crucible

Audit how the Upstairs responded to errors before `ErrorReport`

Open
#857 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
260
Forks
34
Avg merge
2d 1h
Merged PRs (30d)
8

Description

#817 added ErrorReport to the Crucible protocol, and changed the Downstairs so that it would not send Err as part of the Result of all other messages (as part of retrying jobs until they succeed). One known bug resulted from this. It is critical for the correctness of data that jobs be retried until they succeed in the Downstairs, but that PR retained the "abort if a LiveRepair job fails" behaviour of the Downstairs. The Upstairs expected to receive that error and disconnect said Downstairs but this didn't happen anymore, leading to #837.

This issue tracks the work of auditing how the Upstairs previously responded to errors, and change that in the face of ErrorReports.

One possible next step here is to change each Result<T, CrucibleError> that existed previously in the other messages to only T. This would make logic that previously acted on the Err variant a compile error and would help with auditing. Note that we can't straight up commit this change, as this is a major version bump of the protocol and will require thinking about how an Upstairs should have support for the previous messages with Result in them as well as new messages that don't have that.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the Crucible protocol changes from #817 and the failure described in #837, then audit how the Upstairs handled errors from earlier messages. Check each previous Result<T, CrucibleError> and the ErrorReport behavior; done means the Upstairs response is defined for these errors while preserving compatibility with previous and new protocol messages.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
distributed-systems
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.