oxidecomputer / oxidecomputer/omicron
Sled reservoir_size is zero after coming back from NMI
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 572
- Forks
- 97
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 96
Description
The sled BRM42220006 no longer takes any new propolis workload after hitting oxidecomputer/opte#758 (it was booted back up with NMI). Looking at the sled_resource_vmm table, it turns out that the reservoir_size is 0 which explains why nexus thinks this sled has no compute resource.
root@[fd00:1122:3344:105::3]:32221/omicron> select id, serial_number, sled_state, sled_policy, usable_physical_ram, reservoir_size from sled;
id | serial_number | sled_state | sled_policy | usable_physical_ram | reservoir_size
---------------------------------------+---------------+----------------+--------------+---------------------+-----------------
05652dc1-b811-4cac-95e1-d32633f2ba75 | BRM42220026 | decommissioned | expunged | 1086608900096 | 869286281216
0c7011f7-a4bf-4daf-90cc-1c2410103300 | BRM42220057 | active | in_service | 1086608900096 | 869498093568
1efda86b-caef-489f-9792-589d7677e59a | BRM23230018 | decommissioned | expunged | 1086608900096 | 869286281216
2707b587-9c7f-4fb0-a7af-37c3b7a9a0fa | BRM44220011 | active | in_service | 1086608900096 | 869498093568
5f6720b8-8a31-45f8-8c94-8e699218f28b | BRM42220017 | active | in_service | 1086608900096 | 869498093568
71def415-55ad-46b4-ba88-3ca55d7fb287 | BRM42220051 | active | in_service | 1086608900096 | 869498093568
7b862eb6-7f50-4c2f-b9a6-0d12ac913d3c | BRM44220010 | active | in_service | 962054848512 | 765156392960
87c2c4fc-b0c7-4fef-a305-78f0ed265bbc | BRM42220014 | active | in_service | 1086608900096 | 869498093568
a2adea92-b56e-44fc-8a0d-7d63b5fd3b93 | BRM42220031 | active | in_service | 1086608900096 | 869498093568
b886b58a-1e3f-4be1-b9f2-0c2e66c6bc88 | BRM42220006 | active | in_service | 1086608900096 | 0
bd96ef7c-4941-4729-b6f7-5f47feecbc4b | BRM27230045 | active | no_provision | 1086608900096 | 869498093568
db183874-65b5-4263-a1c1-ddb2737ae0e9 | BRM42220009 | active | in_service | 1086608900096 | 869498093568
dd83e75a-1edf-4aa1-89a0-cd8b2091a7cd | BRM42220016 | active | in_service | 824615895040 | 650020651008
f15774c1-b8e5-434f-a493-ec43f96cba06 | BRM44220005 | active | in_service | 1086608900096 | 869498093568
The sled doesn't appear to have any DIMM issues:
BRM42220006 # zoneadm list
global
oxz_ntp_82500cc9-f33d-4d59-9e6e-d70ea6133077
oxz_crucible_c33b5912-9985-43ed-98f2-41297e2b796a
oxz_crucible_0022703b-dcfc-44d4-897a-b42f6f53b433
oxz_crucible_46d1afcc-cc3f-4b17-aafc-054dd4862d15
oxz_crucible_af8a8712-457c-4ea7-a8b6-aecb04761c1b
oxz_crucible_b369e133-485c-4d98-8fee-83542d1fd94d
oxz_crucible_65b3db59-9361-4100-9cee-04e32a8c67d3
oxz_crucible_9b8194ee-917d-4abc-a55c-94cea6cdaea1
oxz_crucible_605be8b9-c652-4a5f-94ca-068ec7a39472
oxz_crucible_12afe1c3-bfe6-4278-8240-91d401347d36
oxz_clickhouse_server_69cce3c6-e957-4e12-b3fb-34f5aa6ecc9f
oxz_clickhouse_keeper_95363b23-6e50-4bf1-9fbe-f16bdfc021c2
oxz_external_dns_edd99650-5df1-4241-815d-253e4ef2399c
oxz_crucible_fffddf56-10ca-4b62-9be3-5b3764a5f682
BRM42220006 # prtconf -m
1036271
BRM42220006 # fmadm faulty
BRM42220006 # fmdump -e
TIME CLASS
fmdump: warning: /var/fm/fmd/errlog is empty
It's unclear why the reservsoir size is not set properly.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by examining the sled_resource_vmm table and the recovery path after oxidecomputer/opte#758, using the reported SQL and sled diagnostics as the baseline. Trace why BRM42220006 has reservoir_size 0 despite usable physical RAM and no DIMM or FMA errors. Done means the cause is identified and the sled's reservoir_size is restored correctly after NMI recovery so it can accept propolis workload.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100