Sled Agent needs to do a better job of letting go of disks

The Sled Agent is responsible for managing both internal and external disks. Internally, it uses these disks for a variety of purposes:

• It stores logs and a variety of files on both disk types
• It manages datasets, and uses these disks for accessing local storage

There are many "consumers" of these disks within the sled agent, e.g.:

• Instance/Probe managers: Store filesystems on U.2 datasets
• Support Bundles: Access Debug datasets
• Update system: Stores zone images on datasets
• Temporary files/storage: Can access datasets on "managed" U.2s
• Dump device management

These consumers should be able to continue using disks, but this relationship is a little hairy in the case of "telling sled agent to stop using a disk".

In the "old version" of the Sled Agent, we had a function named use_only_these_disks, attached to some of our disk-using consumers, such as the InstanceManager. This method was called within omicron_physical_disks_ensure, and acted as following:

• When Sled Agent is told "stop using a disk", it would record to a ledger "I intend to not use disk X anymore"
• Sled Agent would then send an updated view of storage to all subsystems that might be using that disk. This included: the storage monitor (don't use unmanaged U.2s as dump devices), the zone bundler (don't create bundles on unmanaged U.2s), the probe manager, and the instance manager.
• Sled Agent would await responses for these subsystems, acting as an acknowledgement: "This disk is no longer being used".
• When this all completed, Sled Agent could assert: "this disk is now unmanaged, we will not be using it anymore".

However, this was removed in d1d3313d54eabf68ee537bcd0564b4f0f246203a . Now, the API to the sled reconciler is asynchronous, and there is no way to identify when the Sled Agent has relinquished control of the underlying disks. Either way, it's also hard to track down "here are all consumers of a disk" -- a client who originally accesses a disk may subsequently map that managed disk to a dataset, and then to a path, and then to a file -- and it's hard to identify "have all these lingering references to the underlying disk actually been removed".

In the case where we mandate "a disk must be physically detached before it can be unmanaged", this matters less - like an arbitrary disk failure, reads and writes to the disk simply cannot happen. However, in a case where the disk is still present and powered on, it has some unfortunate side-effects: "Unmanaged" disks could still be used, for a long time after the sled has been instructed not to use it.

In the future, we may want to:

1. Make the provenance of disk-based access more clear. Perhaps we could have a Rust structure to help identify "if you're accessing a {disk, dataset, path, file, etc}, it is associated with a reference-counted resource X". This may help for both observability and also identifying "when have the references been cleared"?
2. Restore the message passing mechanism to ask: "please stop using a particular disk", or, "please use only this set of disks", triggering all disk-based access to the old disk to be dropped.
3. Expose some API from the sled agent to identify: "we believe the usage of a disk has been relinquished". This could be explicit, or come from inventory.

Start by reading the Sled Agent disk-management consumers and the asynchronous sled reconciler API, then inspect the removed use_only_these_disks behavior and commit d1d3313d54eabf68ee537bcd0564b4f0f246203a. Map how the instance, probe, support-bundle, update, temporary-storage, and dump-device consumers relinquish disk access. Done requires a decided and implemented way to identify when an unmanaged disk is no longer used.