oxidecomputer / oxidecomputer/omicron
`PUT /omicron-physical-disks` timeout on rack3
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 572
- Forks
- 97
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 96
Description
Today while investigating an issue with a different sled, we briefly enabled the blueprint_executor task. PUT /omicron-physical-disks requests to sled 13 (BRM42220064) were consistently timing out after 60 seconds. The sled-agent log is at /staff/rack3/BRM42220064/2024-10-18/oxide-sled-agent-default.log. We see the dropshot "client disconnected" logs from client timeouts that are consistent with the 60sec timeout error we were seeing from Nexus, like:
john@jeeves /staff/rack3/BRM42220064/2024-10-18 $ grep '\[fd00:1122:3344:10e::3\]:35457' oxide-sled-agent-default.log | looker
21:26:54.440Z INFO SledAgent (dropshot (SledAgent)): accepted connection
file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105
local_addr = [fd00:1122:3344:104::1]:12345
remote_addr = [fd00:1122:3344:10e::3]:35457
21:27:54.483Z WARN SledAgent (dropshot (SledAgent)): request handling cancelled (client disconnected)
file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:887
latency_us = 60040091
local_addr = [fd00:1122:3344:104::1]:12345
method = PUT
remote_addr = [fd00:1122:3344:10e::3]:35457
req_id = efaaabcd-36eb-49e2-881c-4de2f7b08340
uri = /omicron-physical-disks
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the sled-agent log at /staff/rack3/BRM42220064/2024-10-18/oxide-sled-agent-default.log and trace the PUT /omicron-physical-disks request, including the period when blueprint_executor was enabled. Confirm where the 60-second delay occurs and document or fix the cause so the request no longer times out; validate the behavior against the reported sled case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- api, backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100