oxidecomputer / oxidecomputer/omicron

`PUT /omicron-physical-disks` timeout on rack3

Open
#6,904 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Description

Today while investigating an issue with a different sled, we briefly enabled the blueprint_executor task. PUT /omicron-physical-disks requests to sled 13 (BRM42220064) were consistently timing out after 60 seconds. The sled-agent log is at /staff/rack3/BRM42220064/2024-10-18/oxide-sled-agent-default.log. We see the dropshot "client disconnected" logs from client timeouts that are consistent with the 60sec timeout error we were seeing from Nexus, like:

john@jeeves /staff/rack3/BRM42220064/2024-10-18 $ grep '\[fd00:1122:3344:10e::3\]:35457' oxide-sled-agent-default.log  | looker
21:26:54.440Z INFO SledAgent (dropshot (SledAgent)): accepted connection
    file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105
    local_addr = [fd00:1122:3344:104::1]:12345
    remote_addr = [fd00:1122:3344:10e::3]:35457
21:27:54.483Z WARN SledAgent (dropshot (SledAgent)): request handling cancelled (client disconnected)
    file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:887
    latency_us = 60040091
    local_addr = [fd00:1122:3344:104::1]:12345
    method = PUT
    remote_addr = [fd00:1122:3344:10e::3]:35457
    req_id = efaaabcd-36eb-49e2-881c-4de2f7b08340
    uri = /omicron-physical-disks

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the sled-agent log at /staff/rack3/BRM42220064/2024-10-18/oxide-sled-agent-default.log and trace the PUT /omicron-physical-disks request, including the period when blueprint_executor was enabled. Confirm where the 60-second delay occurs and document or fix the cause so the request no longer times out; validate the behavior against the reported sled case.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
api, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.