oxidecomputer / oxidecomputer/omicron

don't let user set target release with unreplicated artifacts

Open
#8,321 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Description

I'm trying an upgrade on a4x2 and tuf_artifact_replication seems not to be fully working, or is going slowly. I was still able to set the target release and proceed with the upgrade. The system behaves fairly well -- it just gets stuck executing a blueprint with something like this:

note: using Nexus URL http://[fd00:1122:3344:101::6]:12221
task: "blueprint_executor"
  configured period: every 1m
  currently executing: iter 83, triggered by a periodic timer firing
    started at 2025-06-11T20:19:47.485Z, running for 26067ms
  last completed activation: iter 82, triggered by a periodic timer firing
    started at 2025-06-11T20:18:47.489Z (86s ago) and ran for 51024ms
    target blueprint: 7ff85197-6be4-482c-8ded-c8b114ca07eb                                                                                                                                                                                                                                                                                                                                                                                                                                                                        
    execution:        enabled                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     
    status:           completed (14 steps)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        
    warning:          at: Deploy sled configs: Failed to put OmicronSledConfig {                                                                                                                                                                                                                                                                                                                                                                                                                                                  
                          generation: Generation(                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 
                              9,                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  
                          ),                                                                   
...
                      } to sled 2d190199-1a3a-419c-8f07-13e00352306e: Error Response: status: 400 Bad Request; headers: {"content-type": "application/json", "x-request-id": "c62efedb-0d2f-47f0-90ef-0edb51c90944", "content-length": "210", "date": "Wed, 11 Jun 2025 20:18:51 GMT"}; value: Error { error_code: None, message: "sled config failed artifact store existence checks: Artifact be6aab2e39fcf5882e94e749ddd394eae45a322deac0528edae8143f9d53fed5 not found", request_id: "c62efedb-0d2f-47f0-90ef-0edb51c90944" } 
    error:            (none)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      


Eventually in at least some cases so far the artifact does show up and then execution succeeds and the upgrade continues. So the system is handling it about as well as it can, but I imagine we want to prevent you from starting an upgrade when the artifacts aren't replicated everywhere.

This is admittedly tricky -- you could add a sled in the middle of an upgrade and that shouldn't stop it. And we probably would need to be able to override this check if we've got some busted sled. But if we just check this at the point where you set the target release, that might be useful.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the target-release setting flow and trace how tuf_artifact_replication reports artifact availability across sleds. Clarify the precondition, behavior when sleds are added during an upgrade, and an override for unavailable sleds; done means an upgrade cannot start with unreplicated artifacts unless explicitly overridden.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
api, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.