oxidecomputer / oxidecomputer/omicron
don't let user set target release with unreplicated artifacts
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 572
- Forks
- 97
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 96
Description
I'm trying an upgrade on a4x2 and tuf_artifact_replication seems not to be fully working, or is going slowly. I was still able to set the target release and proceed with the upgrade. The system behaves fairly well -- it just gets stuck executing a blueprint with something like this:
note: using Nexus URL http://[fd00:1122:3344:101::6]:12221
task: "blueprint_executor"
configured period: every 1m
currently executing: iter 83, triggered by a periodic timer firing
started at 2025-06-11T20:19:47.485Z, running for 26067ms
last completed activation: iter 82, triggered by a periodic timer firing
started at 2025-06-11T20:18:47.489Z (86s ago) and ran for 51024ms
target blueprint: 7ff85197-6be4-482c-8ded-c8b114ca07eb
execution: enabled
status: completed (14 steps)
warning: at: Deploy sled configs: Failed to put OmicronSledConfig {
generation: Generation(
9,
),
...
} to sled 2d190199-1a3a-419c-8f07-13e00352306e: Error Response: status: 400 Bad Request; headers: {"content-type": "application/json", "x-request-id": "c62efedb-0d2f-47f0-90ef-0edb51c90944", "content-length": "210", "date": "Wed, 11 Jun 2025 20:18:51 GMT"}; value: Error { error_code: None, message: "sled config failed artifact store existence checks: Artifact be6aab2e39fcf5882e94e749ddd394eae45a322deac0528edae8143f9d53fed5 not found", request_id: "c62efedb-0d2f-47f0-90ef-0edb51c90944" }
error: (none)
Eventually in at least some cases so far the artifact does show up and then execution succeeds and the upgrade continues. So the system is handling it about as well as it can, but I imagine we want to prevent you from starting an upgrade when the artifacts aren't replicated everywhere.
This is admittedly tricky -- you could add a sled in the middle of an upgrade and that shouldn't stop it. And we probably would need to be able to override this check if we've got some busted sled. But if we just check this at the point where you set the target release, that might be useful.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the target-release setting flow and trace how tuf_artifact_replication reports artifact availability across sleds. Clarify the precondition, behavior when sleds are added during an upgrade, and an override for unavailable sleds; done means an upgrade cannot start with unreplicated artifacts unless explicitly overridden.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- api, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100