oxidecomputer / oxidecomputer/omicron
Consider adding a quota to the crash dataset on the M.2 drives
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 572
- Forks
- 97
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 96
Description
In a recent customer incident all of the debug datasets on the U.2s a sled were filled with core dumps from a flapping service, preventing sled-agent from moving them off of the crash dataset on the M.2 drive. This in turn caused the M.2 drives to be completely filled by core dumps as there is no quota set on that dataset.
BRM44220005 # zfs list -o name,used,avail,quota | egrep -e NAME -e 'oxi_109e5273.*/'
NAME USED AVAIL QUOTA
oxi_109e5273-7221-46ec-9415-620ce87f0777/backing 96K 722G none
oxi_109e5273-7221-46ec-9415-620ce87f0777/cluster 160K 722G none
oxi_109e5273-7221-46ec-9415-620ce87f0777/config 148K 722G none
oxi_109e5273-7221-46ec-9415-620ce87f0777/crash 120K 722G none
oxi_109e5273-7221-46ec-9415-620ce87f0777/debug 1.35G 98.6G 100G
oxi_109e5273-7221-46ec-9415-620ce87f0777/install 1.03G 722G none
oxi_109e5273-7221-46ec-9415-620ce87f0777/update 100K 20.0G 20G
We should consider whether to add a quota to the crash dataset, or any other quota-less datasets on these drives.
Related to https://github.com/oxidecomputer/omicron/issues/7875
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing sled-agent's handling of the crash dataset and reviewing how the quota-less ZFS datasets in the report are created or configured. Determine which datasets need quotas and what limits preserve space for dataset movement; done means an agreed policy is implemented and verified against the reported failure mode.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- infrastructure
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 42/100