How flexible should sled-agent be in terms of environment / gimlet support?

Today, you can run sled-agent on a real gimlet or a variety of other underlying hardware (or VMs), and it detects at runtime whether it thinks it's running on a real gimlet. This can lead to some odd failure modes; for example, if sled-agent runs on a real gimlet that has not been programmed (or reading the VPD has failed), sled_hardware::is_gimlet() will return false, which will lead to underlay::find_chelsio_nics() to look for net0/net1 devices instead of the real chelsio devices.

I'm not sure what action should be taken here, exactly. Maybe sled-agent should have more compile-time configuration and less runtime configuration? Maybe the runtime configuration is okay, at least for the near term?

Start with sled-hardware/src/illumos/mod.rs at sled_hardware::is_gimlet() and sled-hardware/src/underlay.rs at underlay::find_chelsio_nics(). Trace how sled-agent selects its environment and hardware paths, then review the issue discussion for constraints. Done means an agreed configuration approach and a defined resolution for the unprogrammed-gimlet and VPD-read-failure cases.