oxidecomputer / oxidecomputer/omicron

during local storage disk delete, sled agent sometimes returns `dataset is busy`

Open
#10,818 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Description

The following was reported by @askfongjojo during testing of increased reservoir size on dogfood, but I don't (yet) think the two are related.

A terraform destroy was in progress, but this error was seen:

21:14:36.782Z INFO 6a11df2d-df48-415e-a5d4-bd591d593c96 (dropshot_external): request completed
    error_message_external = Internal Server Error
    error_message_internal = saga ACTION error at node "delete_local_storage": failed to delete local storage: failed at attempt 1: progenitor API operation failed: Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "62b5e974-67a0-46df-8485-713229622540", "content-length": "124", "date": "Mon, 13 Jul 2026 21:14:36 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "62b5e974-67a0-46df-8485-713229622540" }
    file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:862
    latency_us = 13591226
    local_addr = 172.30.2.10:443
    method = DELETE
    remote_addr = 172.20.16.14:59534
    req_id = 09327793-876a-45e7-8910-c228f6b4fb35
    response_code = 500
    uri = https://oxide.sys.rack2.eng.oxide.computer/v1/disks/74f3f328-067a-490a-b071-8f6c46d7b302

Checking later, I did find the disk in question was deleted with a few minutes of this request:

[BRM42220014]root@oxz_switch1:~# omdb db disks info 74f3f328-067a-490a-b071-8f6c46d7b302
HOST_SERIAL DISK_NAME                                    INSTANCE_NAME PROPOLIS_ZONE DISK_STATE
-           deleted-74f3f328-067a-490a-b071-8f6c46d7b302 -             -             destroyed
DISK_NAME                                    TIME_CREATED                   TIME_DELETED                   ALLOCATION_TYPE DATASET_ID                           POOL_ID                              SLED_ID                              DATASET_SIZE
deleted-74f3f328-067a-490a-b071-8f6c46d7b302 2026-07-13 21:09:31.416573 UTC 2026-07-13 21:16:09.058592 UTC unencrypted     152a4334-c136-41f0-969a-4fe87d628bc3 416fd29e-d3b5-4fdf-8101-d0d163fa0706 0c7011f7-a4bf-4daf-90cc-1c2410103300 57357107200

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the delete_local_storage saga action and the sled agent's disk DELETE path using the logged request ID and dataset is busy response. Inspect the timing around the later successful deletion and any relevant tests; done means the intermittent delete failure is understood and covered by regression testing.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
api, backend, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.