oxidecomputer / oxidecomputer/omicron
during local storage disk delete, sled agent sometimes returns `dataset is busy`
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 572
- Forks
- 97
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 96
Description
The following was reported by @askfongjojo during testing of increased reservoir size on dogfood, but I don't (yet) think the two are related.
A terraform destroy was in progress, but this error was seen:
21:14:36.782Z INFO 6a11df2d-df48-415e-a5d4-bd591d593c96 (dropshot_external): request completed
error_message_external = Internal Server Error
error_message_internal = saga ACTION error at node "delete_local_storage": failed to delete local storage: failed at attempt 1: progenitor API operation failed: Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "62b5e974-67a0-46df-8485-713229622540", "content-length": "124", "date": "Mon, 13 Jul 2026 21:14:36 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "62b5e974-67a0-46df-8485-713229622540" }
file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:862
latency_us = 13591226
local_addr = 172.30.2.10:443
method = DELETE
remote_addr = 172.20.16.14:59534
req_id = 09327793-876a-45e7-8910-c228f6b4fb35
response_code = 500
uri = https://oxide.sys.rack2.eng.oxide.computer/v1/disks/74f3f328-067a-490a-b071-8f6c46d7b302
Checking later, I did find the disk in question was deleted with a few minutes of this request:
[BRM42220014]root@oxz_switch1:~# omdb db disks info 74f3f328-067a-490a-b071-8f6c46d7b302
HOST_SERIAL DISK_NAME INSTANCE_NAME PROPOLIS_ZONE DISK_STATE
- deleted-74f3f328-067a-490a-b071-8f6c46d7b302 - - destroyed
DISK_NAME TIME_CREATED TIME_DELETED ALLOCATION_TYPE DATASET_ID POOL_ID SLED_ID DATASET_SIZE
deleted-74f3f328-067a-490a-b071-8f6c46d7b302 2026-07-13 21:09:31.416573 UTC 2026-07-13 21:16:09.058592 UTC unencrypted 152a4334-c136-41f0-969a-4fe87d628bc3 416fd29e-d3b5-4fdf-8101-d0d163fa0706 0c7011f7-a4bf-4daf-90cc-1c2410103300 57357107200
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the delete_local_storage saga action and the sled agent's disk DELETE path using the logged request ID and dataset is busy response. Inspect the timing around the later successful deletion and any relevant tests; done means the intermittent delete failure is understood and covered by regression testing.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- api, backend, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 42/100