oxidecomputer / oxidecomputer/crucible

Agent failed to delete a region

Open
#1,666 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
260
Forks
34
Avg merge
2d 1h
Merged PRs (30d)
8

Description

Crucible agent had trouble deleting a newly created region:

00:49:47.053Z INFO crucible-agent (dropshot): request completed
    latency_us = 68
    local_addr = [fd00:1122:3344:106::4]:32345
    method = GET
    remote_addr = [fd00:1122:3344:10f::3]:37967
    req_id = 386ab886-b906-4b58-ad9b-09744b267f3a
    response_code = 200
    uri = /crucible/0/regions/623874c5-efc6-4b87-aa1d-cc496e46a12e
00:49:47.395Z INFO crucible-agent (worker): region files created ok
    region = 83ba6068-1df4-49ab-972b-67d36a1aa029
00:49:47.395Z INFO crucible-agent (datafile): region 83ba6068-1df4-49ab-972b-67d36a1aa029 state: Requested -> Created
00:49:47.395Z INFO crucible-agent (worker): applying SMF actions post create...
00:49:47.456Z INFO crucible-agent (worker): disabling downstairs instance: downstairs-623874c5-efc6-4b87-aa1d-cc496e46a12e (instance states: (Some(Online), None))
00:49:47.481Z INFO crucible-agent (worker): creating missing downstairs instance downstairs-83ba6068-1df4-49ab-972b-67d36a1aa029
00:49:47.485Z INFO crucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-83ba6068-1df4-49ab-972b-67d36a1aa029
00:49:47.492Z INFO crucible-agent (worker): creating config property group
00:49:47.496Z INFO crucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-83ba6068-1df4-49ab-972b-67d36a1aa029
00:49:47.497Z INFO crucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/83ba6068-1df4-49ab-972b-67d36a1aa029
00:49:47.497Z INFO crucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
00:49:47.497Z INFO crucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:106::4
00:49:47.497Z INFO crucible-agent (worker): commit
00:49:47.499Z INFO crucible-agent (worker): ok!
00:49:47.507Z INFO crucible-agent (worker): SMF ok!
00:49:47.507Z INFO crucible-agent (worker): applying SMF actions before removal...
00:49:47.534Z INFO crucible-agent (worker): disabling downstairs instance: downstairs-623874c5-efc6-4b87-aa1d-cc496e46a12e (instance states: (Some(Online), Some(Disabled)))
00:49:47.538Z INFO crucible-agent (worker): SMF ok!
00:49:47.566Z INFO crucible-agent (worker): deleting zfs dataset "oxp_9b1be26b-6af2-4ec4-8074-9aafadccf73d/crucible/regions/623874c5-efc6-4b87-aa1d-cc496e46a12e"
    region = 623874c5-efc6-4b87-aa1d-cc496e46a12e
00:49:47.610Z INFO crucible-agent (dropshot): request completed
    latency_us = 155
    local_addr = [fd00:1122:3344:106::4]:32345
    method = GET
    remote_addr = [fd00:1122:3344:10e::3]:50908
    req_id = f372ce6d-7510-49e8-90cb-d14814f2d5fb
    response_code = 200
    uri = /crucible/0/regions/623874c5-efc6-4b87-aa1d-cc496e46a12e
00:49:47.623Z ERRO crucible-agent (worker): zfs dataset oxp_9b1be26b-6af2-4ec4-8074-9aafadccf73d/crucible/regions/623874c5-efc6-4b87-aa1d-cc496e46a12e delete attempt 0 failed: out: err:cannot unmount '/data/regions/623874c5-efc6-4b87-aa1d-cc496e46a12e': Device busy
    region = 623874c5-efc6-4b87-aa1d-cc496e46a12e
00:49:50.965Z INFO crucible-agent (dropshot): request completed
    latency_us = 23476
    local_addr = [fd00:1122:3344:106::4]:32345
    method = GET
    remote_addr = [fd00:1122:3344:10f::3]:37967
    req_id = f215d40e-d411-495b-804a-7d4a6e6e1d30
    response_code = 200
    uri = /crucible/0/regions/623874c5-efc6-4b87-aa1d-cc496e46a12e

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No source file or test is named. Start with the agent's region-removal path corresponding to the deleting zfs dataset log and trace how the downstairs instance is stopped before deletion. Done means a newly created region can be deleted without the reported Device busy failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.