Cannot start instance because `507 Insufficient Storage`

On dogfood, while attempting to run the system out of resources, I encountered an unexpected failure.

I seem to have enough CPU, Memory, and Storage resources available:

alan@atrium:many-instances$ oxide utilization
{
  "capacity": {
    "cpus": 960,
    "memory": 8367025789338,
    "storage": 38654705664000
  },
  "provisioned": {
    "cpus": 723,
    "memory": 2057289334784,
    "storage": 16909286244352
  }
}

However, when I attempt to start an instance:

alan@atrium:many-instances$ oxide instance start --project alan --instance garrett-inst-100
error
Error Response: status: 507 Insufficient Storage; headers: {"content-type": "application/json", "x-request-id": "3665f612-8ecf-4500-8cd0-6304a80c5ec2", "content-length": "177", "date": "Tue, 20 Feb 2024 00:15:46 GMT"}; value: Error { error_code: Some("InsufficientCapacity"), message: "Insufficient capacity: No sleds can fit the requested instance", request_id: "3665f612-8ecf-4500-8cd0-6304a80c5ec2" }

My already created instance has a disk that is also already created:

alan@atrium:many-instances$ oxide instance view --project alan --instance garrett-inst-100
{
  "description": "loop host",
  "hostname": "fff-100",
  "id": "f8be046f-8bab-45f6-8c35-58de24b33099",
  "memory": 4294967296,
  "name": "garrett-inst-100",
  "ncpus": 64,
  "project_id": "759beaf2-517d-4d24-bc17-1eed69bc8801",
  "run_state": "stopped",
  "time_created": "2024-02-20T00:05:02.314662Z",
  "time_modified": "2024-02-20T00:05:02.314662Z",
  "time_run_state_updated": "2024-02-20T00:05:02.314662Z"
}

Start by reproducing the failure with oxide utilization and oxide instance start, then compare the reported capacity with the instance details from oxide instance view. Trace the InsufficientCapacity response and the “No sleds can fit” condition. Done means the reported resource state and start behavior are consistent, or the failure is explained by the relevant capacity constraint.