Sled agent hung on disabling `fmd` service.

Spinning up an omicron development environment on my workstation hung at the following place. Sled agent was running, but not making any progress. It had brought up the switch zone, but had not brought up any other zones.

https://github.com/oxidecomputer/omicron/blob/194889b956abbb3e01ce25b11b733c02598c3215/sled-agent/src/backing_fs.rs#L157-L161

We were able to see this on the host OS as follows.

ry@masaka:~/src/omicron$ pgrep -lf sled-agent
15742 /opt/oxide/sled-agent/sled-agent run /opt/oxide/sled-agent/pkg/config.toml
15741 ctrun -l child -o noorphan,regent /opt/oxide/sled-agent/sled-agent run /opt/oxi

ry@masaka:~/src/omicron$ pfexec ptree 15742
15741  ctrun -l child -o noorphan,regent /opt/oxide/sled-agent/sled-agent run /opt/oxi
  15742  /opt/oxide/sled-agent/sled-agent run /opt/oxide/sled-agent/pkg/config.toml
    15903  /usr/sbin/svcadm disable -s -t svc:/system/fmd:default

Looking at the fmd service, we found it in maintenance mode.

ry@masaka:~/src/omicron$ svcs | grep fmd
maintenance    17:05:33 svc:/system/fmd:default

Clearing the service with svcadm clear fmd resulted in sled agent springing to life. At this point, it had been stuck for over an hour as I went to work on other things. The control plane came up completely and appears to be functioning normally after getting unstuck.

I looked at the logs for fmd with cat $(svcs -L fmd) the contained this

[ Sep 18 22:28:05 Method "start" exited with status 0. ]
[ Oct 10 15:59:10 Stopping because service disabled. ]
[ Oct 10 15:59:10 Executing stop method (:kill). ]
[ Oct 10 15:59:11 Enabled. ]
[ Oct 10 15:59:11 Executing start method ("/usr/lib/fm/fmd/fmd"). ]
[ Oct 10 16:00:33 Method "start" exited with status 0. ]
[ Oct 10 17:04:33 Stopping because service disabled. ]
[ Oct 10 17:04:33 Executing stop method (:kill). ]
[ Oct 10 17:05:33 Method or service exit timed out.  Killing contract 61062. ]

Start with sled-agent/src/backing_fs.rs at lines 157-161 and inspect how the fmd service-disable command is handled. Reproduce the maintenance-mode case described in the issue and trace why sled-agent waits there. Done means the agent no longer remains hung while disabling fmd and the development environment can continue starting.