oxidecomputer / oxidecomputer/omicron

Consider adding a quota to the crash dataset on the M.2 drives

Open
#7,882 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

customer storage
Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Description

In a recent customer incident all of the debug datasets on the U.2s a sled were filled with core dumps from a flapping service, preventing sled-agent from moving them off of the crash dataset on the M.2 drive. This in turn caused the M.2 drives to be completely filled by core dumps as there is no quota set on that dataset.

BRM44220005 # zfs list -o name,used,avail,quota  | egrep -e NAME -e 'oxi_109e5273.*/'
NAME                                                 USED  AVAIL  QUOTA
oxi_109e5273-7221-46ec-9415-620ce87f0777/backing      96K   722G   none
oxi_109e5273-7221-46ec-9415-620ce87f0777/cluster     160K   722G   none
oxi_109e5273-7221-46ec-9415-620ce87f0777/config      148K   722G   none
oxi_109e5273-7221-46ec-9415-620ce87f0777/crash       120K   722G   none
oxi_109e5273-7221-46ec-9415-620ce87f0777/debug      1.35G  98.6G   100G
oxi_109e5273-7221-46ec-9415-620ce87f0777/install    1.03G   722G   none
oxi_109e5273-7221-46ec-9415-620ce87f0777/update      100K  20.0G    20G

We should consider whether to add a quota to the crash dataset, or any other quota-less datasets on these drives.

Related to https://github.com/oxidecomputer/omicron/issues/7875

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing sled-agent's handling of the crash dataset and reviewing how the quota-less ZFS datasets in the report are created or configured. Determine which datasets need quotas and what limits preserve space for dataset movement; done means an agreed policy is implemented and verified against the reported failure mode.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
infrastructure
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.