Sled has a reservoir size of 0 after reboot

We have seen four sleds in rack3 that have come up after a recent restart with a reservoir size of 0 set in the database.

root@[fd00:1122:3344:116::3]:32221/omicron> select id, serial_number, reservoir_size, time_created, time_modified from sled where reservoir_size = 0;
                   id                  | serial_number | reservoir_size |         time_created          |         time_modified
---------------------------------------+---------------+----------------+-------------------------------+--------------------------------
  2c33e8ef-ae07-4191-804d-52a1840ce53c | BRM42220042   |              0 | 2023-09-02 13:07:22.168451+00 | 2026-08-12 15:39:45.228129+00
  857bd97b-8147-4d26-813b-d80ab8f36cd5 | BRM42220083   |              0 | 2023-09-02 13:07:20.885306+00 | 2026-08-12 17:45:59.820146+00
  eaad4483-ea93-48d1-99d3-bf43267b9eb7 | BRM42220015   |              0 | 2023-09-02 13:07:20.852863+00 | 2026-08-12 21:23:28.448887+00
  fac5180b-232a-489e-b7c5-313a0742d1cc | BRM42220054   |              0 | 2023-09-02 13:07:20.82203+00  | 2026-08-12 16:04:53.65634+00

All four of these sleds have sled-agent logs that look like:

{"msg":"Calculated eligible VMM reservoir size","v":0,"name":"SledAgent","level":30,"time":"1986-12-28T00:01:11.231465519Z","hostname":"BRM42220015","pid":665,"component":"MemoryReservations","file":"sled-hardware/src/lib.rs:154","control_plane_earmark_bytes":"47244640256","max_page_t_bytes":"31834245120","physical_ram_bytes":"1086608900096","vmm_eligible":"1007530014720"}
{"msg":"91% of 1007530014720 VMM eligible ram = 916852313395 bytes)","v":0,"name":"SledAgent","level":30,"time":"1986-12-28T00:01:11.23220472Z","hostname":"BRM42220015","pid":665,"component":"VmmReservoirManager","sled_id":"eaad4483-ea93-48d1-99d3-bf43267b9eb7","component":"SledAgent","file":"sled-agent/src/vmm_reservoir.rs:258"}
{"msg":"Setting reservoir size to 874378 MiB bytes","v":0,"name":"SledAgent","level":30,"time":"1986-12-28T00:01:11.232244165Z","hostname":"BRM42220015","pid":665,"component":"VmmReservoirManager","sled_id":"eaad4483-ea93-48d1-99d3-bf43267b9eb7","component":"SledAgent","file":"sled-agent/src/vmm_reservoir.rs:266"}
{"msg":"Retrieved SledAgentInfo from Nexus: SledAgentInfo { baseboard: Baseboard { part: \"913-0000019\", revision: 6, serial: \"BRM42220015\" }, cpu_family: AmdMilan, decommissioned: false, generation: Generation(27), repo_depot_port: 12348, reservoir_size: ByteCount(916851785728), role: Gimlet, sa_address: \"[fd00:1122:3344:109::1]:12345\", usable_hardware_threads: 128, usable_physical_ram: ByteCount(1086608900096) }","v":0,"name":"SledAgent","level":30,"time":"1986-12-28T00:01:11.256005206Z","hostname":"BRM42220015","pid":665,"component":"NexusNotifierTask","sled_id":"eaad4483-ea93-48d1-99d3-bf43267b9eb7","component":"SledAgent","file":"sled-agent/src/nexus.rs:500"}
{"msg":"Failed to setup VMM reservoir","v":0,"name":"SledAgent","level":50,"time":"2026-08-12T21:27:06.348569867Z","hostname":"BRM42220015","pid":665,"component":"VmmReservoirManager","sled_id":"eaad4483-ea93-48d1-99d3-bf43267b9eb7","component":"SledAgent","file":"sled-agent/src/vmm_reservoir.rs:191","error":"Failed to create reservoir: Failed to resize reservoir: Not enough space (os error 12)"}

We do know that earlier in the day after a reboot we did successfully allocate the reservoir:

{"msg":"Calculated eligible VMM reservoir size","v":0,"name":"SledAgent","level":30,"time":"1986-12-28T00:00:59.461675821Z","hostname":"BRM42220015","pid":666,"component":"MemoryReservations","file":"sled-hardware/src/lib.rs:154","control_plane_earmark_bytes":"47244640256","max_page_t_bytes":"31834245120","physical_ram_bytes":"1086608900096","vmm_eligible":"1007530014720"}
{"msg":"91% of 1007530014720 VMM eligible ram = 916852313395 bytes)","v":0,"name":"SledAgent","level":30,"time":"1986-12-28T00:00:59.462365518Z","hostname":"BRM42220015","pid":666,"component":"VmmReservoirManager","sled_id":"eaad4483-ea93-48d1-99d3-bf43267b9eb7","component":"SledAgent","file":"sled-agent/src/vmm_reservoir.rs:258"}
{"msg":"Setting reservoir size to 874378 MiB bytes","v":0,"name":"SledAgent","level":30,"time":"1986-12-28T00:00:59.462378383Z","hostname":"BRM42220015","pid":666,"component":"VmmReservoirManager","sled_id":"eaad4483-ea93-48d1-99d3-bf43267b9eb7","component":"SledAgent","file":"sled-agent/src/vmm_reservoir.rs:266"}
{"msg":"Retrieved SledAgentInfo from Nexus: SledAgentInfo { baseboard: Baseboard { part: \"913-0000019\", revision: 6, serial: \"BRM42220015\" }, cpu_family: AmdMilan, decommissioned: false, generation: Generation(25), repo_depot_port: 12348, reservoir_size: ByteCount(916851785728), role: Gimlet, sa_address: \"[fd00:1122:3344:109::1]:12345\", usable_hardware_threads: 128, usable_physical_ram: ByteCount(1086608900096) }","v":0,"name":"SledAgent","level":30,"time":"1986-12-28T00:00:59.484793759Z","hostname":"BRM42220015","pid":666,"component":"NexusNotifierTask","sled_id":"eaad4483-ea93-48d1-99d3-bf43267b9eb7","component":"SledAgent","file":"sled-agent/src/nexus.rs:500"}
{"msg":"Finished setting reservoir size to 874378 MiB bytes","v":0,"name":"SledAgent","level":30,"time":"2026-08-12T14:37:53.242773137Z","hostname":"BRM42220015","pid":666,"component":"VmmReservoirManager","sled_id":"eaad4483-ea93-48d1-99d3-bf43267b9eb7","component":"SledAgent","file":"sled-agent/src/vmm_reservoir.rs:270"}

After the failure we see the following:

BRM42220015 # mdb -ke '::memstat'
Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                   15050243             58790    6%
Boot pages                     11                 0    0%
ZFS File Data            23488784             91753    9%
VMM Memory              214695936            838656   81%
Anon                      6306679             24635    2%
Exec and libs               88068               344    0%
Page cache                   6815                26    0%
Free (cachelist)           313123              1223    0%
Free (freelist)           5333668             20834    2%

Total                   265283327           1036262
Physical                265283325           1036262

We do not currently see this on rack2, but we have a sled on Berlin that has reproduced this behavior multiple times as noted by @leftwo .

BRM44220007 # grep reservoir $(svcs -L sled-agent) | looker
00:00:52.446Z INFO SledAgent (MemoryReservations): Calculated eligible VMM reservoir size
    control_plane_earmark_bytes = 47244640256
    file = sled-hardware/src/lib.rs:154
    max_page_t_bytes = 31834245120
    physical_ram_bytes = 1086608900096
    vmm_eligible = 1007530014720
00:00:52.446Z INFO SledAgent: 91% of 1007530014720 VMM eligible ram = 916852313395 bytes)
    file = sled-agent/src/vmm_reservoir.rs:258
    sled_id = 8f335797-3472-461d-b106-a17e8dc7f255
00:00:52.447Z INFO SledAgent: Setting reservoir size to 874378 MiB bytes
    file = sled-agent/src/vmm_reservoir.rs:266
    sled_id = 8f335797-3472-461d-b106-a17e8dc7f255
00:00:52.491Z INFO SledAgent: Retrieved SledAgentInfo from Nexus: SledAgentInfo { baseboard: Baseboard { part: "913-0000019", revision: 6, serial: "BRM44220007" }, cpu_family: AmdMilan, decommissioned: false, generation: Generation(2), repo_depot_port: 12348, reservoir_size: ByteCount(0), role: Gimlet, sa_address: "[fd35:e7fa:ed75:103::1]:12345", usable_hardware_threads: 128, usable_physical_ram: ByteCount(1086608900096) }
    file = sled-agent/src/nexus.rs:500
    sled_id = 8f335797-3472-461d-b106-a17e8dc7f255
20:28:38.000Z ERRO SledAgent: Failed to setup VMM reservoir
    error = Failed to create reservoir: Failed to resize reservoir: Not enough space (os error 12)
    file = sled-agent/src/vmm_reservoir.rs:191
    sled_id = 8f335797-3472-461d-b106-a17e8dc7f255

Specifically we are failing at https://github.com/oxidecomputer/omicron/blob/main/illumos-utils/src/vmm_reservoir.rs#L50 from our current understanding.