oxidecomputer / oxidecomputer/omicron

Sled reservoir_size is zero after coming back from NMI

Open
#8,257 8 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Description

The sled BRM42220006 no longer takes any new propolis workload after hitting oxidecomputer/opte#758 (it was booted back up with NMI). Looking at the sled_resource_vmm table, it turns out that the reservoir_size is 0 which explains why nexus thinks this sled has no compute resource.

root@[fd00:1122:3344:105::3]:32221/omicron> select id, serial_number, sled_state, sled_policy, usable_physical_ram, reservoir_size from sled;
                   id                  | serial_number |   sled_state   | sled_policy  | usable_physical_ram | reservoir_size
---------------------------------------+---------------+----------------+--------------+---------------------+-----------------
  05652dc1-b811-4cac-95e1-d32633f2ba75 | BRM42220026   | decommissioned | expunged     |       1086608900096 |   869286281216
  0c7011f7-a4bf-4daf-90cc-1c2410103300 | BRM42220057   | active         | in_service   |       1086608900096 |   869498093568
  1efda86b-caef-489f-9792-589d7677e59a | BRM23230018   | decommissioned | expunged     |       1086608900096 |   869286281216
  2707b587-9c7f-4fb0-a7af-37c3b7a9a0fa | BRM44220011   | active         | in_service   |       1086608900096 |   869498093568
  5f6720b8-8a31-45f8-8c94-8e699218f28b | BRM42220017   | active         | in_service   |       1086608900096 |   869498093568
  71def415-55ad-46b4-ba88-3ca55d7fb287 | BRM42220051   | active         | in_service   |       1086608900096 |   869498093568
  7b862eb6-7f50-4c2f-b9a6-0d12ac913d3c | BRM44220010   | active         | in_service   |        962054848512 |   765156392960
  87c2c4fc-b0c7-4fef-a305-78f0ed265bbc | BRM42220014   | active         | in_service   |       1086608900096 |   869498093568
  a2adea92-b56e-44fc-8a0d-7d63b5fd3b93 | BRM42220031   | active         | in_service   |       1086608900096 |   869498093568
  b886b58a-1e3f-4be1-b9f2-0c2e66c6bc88 | BRM42220006   | active         | in_service   |       1086608900096 |              0
  bd96ef7c-4941-4729-b6f7-5f47feecbc4b | BRM27230045   | active         | no_provision |       1086608900096 |   869498093568
  db183874-65b5-4263-a1c1-ddb2737ae0e9 | BRM42220009   | active         | in_service   |       1086608900096 |   869498093568
  dd83e75a-1edf-4aa1-89a0-cd8b2091a7cd | BRM42220016   | active         | in_service   |        824615895040 |   650020651008
  f15774c1-b8e5-434f-a493-ec43f96cba06 | BRM44220005   | active         | in_service   |       1086608900096 |   869498093568

The sled doesn't appear to have any DIMM issues:

BRM42220006 # zoneadm list
global
oxz_ntp_82500cc9-f33d-4d59-9e6e-d70ea6133077
oxz_crucible_c33b5912-9985-43ed-98f2-41297e2b796a
oxz_crucible_0022703b-dcfc-44d4-897a-b42f6f53b433
oxz_crucible_46d1afcc-cc3f-4b17-aafc-054dd4862d15
oxz_crucible_af8a8712-457c-4ea7-a8b6-aecb04761c1b
oxz_crucible_b369e133-485c-4d98-8fee-83542d1fd94d
oxz_crucible_65b3db59-9361-4100-9cee-04e32a8c67d3
oxz_crucible_9b8194ee-917d-4abc-a55c-94cea6cdaea1
oxz_crucible_605be8b9-c652-4a5f-94ca-068ec7a39472
oxz_crucible_12afe1c3-bfe6-4278-8240-91d401347d36
oxz_clickhouse_server_69cce3c6-e957-4e12-b3fb-34f5aa6ecc9f
oxz_clickhouse_keeper_95363b23-6e50-4bf1-9fbe-f16bdfc021c2
oxz_external_dns_edd99650-5df1-4241-815d-253e4ef2399c
oxz_crucible_fffddf56-10ca-4b62-9be3-5b3764a5f682
BRM42220006 # prtconf -m
1036271
BRM42220006 # fmadm faulty
BRM42220006 # fmdump -e
TIME                 CLASS
fmdump: warning: /var/fm/fmd/errlog is empty

It's unclear why the reservsoir size is not set properly.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by examining the sled_resource_vmm table and the recovery path after oxidecomputer/opte#758, using the reported SQL and sled diagnostics as the baseline. Trace why BRM42220006 has reservoir_size 0 despite usable physical RAM and no DIMM or FMA errors. Done means the cause is identified and the sled's reservoir_size is restored correctly after NMI recovery so it can accept propolis workload.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.