instance stop blocked on crucible

This is an example of oxidecomputer/propolis#371. It was seen on the colo rack. The log files are on catacomb at /net/catacomb/data/staff/augustus/rack3/BRM44220009/logs/propolis-logs.tar.gz.

(I didn't try to make sense of the log messages beyond triaging into propolis#371, so it's possible this is a known issue, but I wanted to dump state before I have to step out somewhat early today.)

Here's the log from just before the Stop request:

[2023-10-06T14:24:48.619209332Z]  WARN: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: [1] Downstairs not responding, take offline
[2023-10-06T14:24:48.61928778Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: [1] e4a0216e-53b8-4e84-9db0-82a6edb24820 Gone missing, transition from Active to Offline
[2023-10-06T14:24:48.619304813Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: [1] e4a0216e-53b8-4e84-9db0-82a6edb24820 connection to [fd00:1122:3344:11b::a]:19001 closed (looper=1)
[2023-10-06T14:24:48.61932443Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: [1] [fd00:1122:3344:11b::a]:19001 task reports connection:false
[2023-10-06T14:24:48.619337635Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: e4a0216e-53b8-4e84-9db0-82a6edb24820 Active Offline Active
[2023-10-06T14:24:48.619350429Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: [1] [fd00:1122:3344:11b::a]:19001 task reports offline
[2023-10-06T14:24:48.619362562Z]  WARN: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: [1] pm_task rx.recv() is None
[2023-10-06T14:24:49.620592286Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: [1] connecting to [fd00:1122:3344:11b::a]:19001 (looper=1)
[2023-10-06T14:24:59.622167208Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
[2023-10-06T14:25:10.625265821Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
[2023-10-06T14:25:21.627356645Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
[2023-10-06T14:25:32.63040417Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
[2023-10-06T14:25:43.63290518Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
[2023-10-06T14:25:54.635836403Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
[2023-10-06T14:26:05.637979175Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
[2023-10-06T14:26:16.640675786Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
[2023-10-06T14:26:27.643529204Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
[2023-10-06T14:26:38.645502691Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
[2023-10-06T14:26:39.646892515Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: [1] connecting to [fd00:1122:3344:11b::a]:19001 (looper=1)
[2023-10-06T14:26:49.648516882Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
[2023-10-06T14:27:00.651430933Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
[2023-10-06T14:27:11.654106021Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
[2023-10-06T14:27:22.657018788Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
[2023-10-06T14:27:33.659935093Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
[2023-10-06T14:27:44.662770516Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
[2023-10-06T14:27:55.665523845Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
[2023-10-06T14:28:06.668646586Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
[2023-10-06T14:28:17.670662457Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
[2023-10-06T14:28:28.673831553Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
[2023-10-06T14:28:29.67426349Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: [1] connecting to [fd00:1122:3344:11b::a]:19001 (looper=1)
[2023-10-06T14:28:39.676497843Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
[2023-10-06T14:28:50.678461832Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
[2023-10-06T14:29:01.681383345Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
[2023-10-06T14:29:12.683744388Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
[2023-10-06T14:29:23.686602374Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
[2023-10-06T14:29:34.689523739Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
[2023-10-06T14:29:45.69246981Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
[2023-10-06T14:29:56.695310272Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
[2023-10-06T14:30:07.698161372Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
[2023-10-06T14:30:18.700102099Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
[2023-10-06T14:30:19.701559551Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: [1] connecting to [fd00:1122:3344:11b::a]:19001 (looper=1)
[2023-10-06T14:30:29.702737261Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
[2023-10-06T14:30:40.70531659Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
[2023-10-06T14:30:51.708150375Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
[2023-10-06T14:31:02.710905936Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
[2023-10-06T14:31:13.713354124Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
[2023-10-06T14:31:24.716034582Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
[2023-10-06T14:31:35.718721294Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
[2023-10-06T14:31:46.720551522Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
[2023-10-06T14:31:57.723913383Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
[2023-10-06T14:32:08.726791003Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
[2023-10-06T14:32:09.728149025Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: [1] connecting to [fd00:1122:3344:11b::a]:19001 (looper=1)
[2023-10-06T14:32:19.729291089Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
[2023-10-06T14:32:30.73236215Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
[2023-10-06T14:32:41.735252017Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
[2023-10-06T14:32:52.738471157Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
[2023-10-06T14:33:03.740542923Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
[2023-10-06T14:33:14.743844806Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
[2023-10-06T14:33:25.746440434Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
[2023-10-06T14:33:36.749073142Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
[2023-10-06T14:33:47.752180058Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
[2023-10-06T14:33:52.148715711Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: [1] e4a0216e-53b8-4e84-9db0-82a6edb24820 looper connected (looper=1)
[2023-10-06T14:33:52.148794421Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: [1] Proc runs for [fd00:1122:3344:11b::a]:19001 in state Offline
[2023-10-06T14:33:52.148814098Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: [1] client re-new 0 jobs since flush 33619 (=downstairs)
[2023-10-06T14:33:52.148883069Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: [1] upstairs guest_io_ready=TRUE, promote! session 5eb7460c-1e82-4594-bcda-326a6cf20de3
[2023-10-06T14:33:52.149127953Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: [1] downstairs client at [fd00:1122:3344:11b::a]:19001 has UUID 80cfef8a-4350-4b24-8f54-c083b6e8f085
[2023-10-06T14:33:52.149147921Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: [1] Got region def RegionDefinition { block_size: 512, extent_size: Block { value: 131072, shift: 9 }, extent_count: 48, uuid: 80cfef8a-4350-4b24-8f54-c083b6e8f085, encrypted: true, database_read_version: 1, database_write_version: 1 }
[2023-10-06T14:33:52.149163791Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: Returning client:1 UUID:80cfef8a-4350-4b24-8f54-c083b6e8f085 matches
[2023-10-06T14:33:52.149177667Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: [1] send last flush ID to this DS: 33619
[2023-10-06T14:33:52.149190301Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: e4a0216e-53b8-4e84-9db0-82a6edb24820 Active Offline Active
[2023-10-06T14:33:52.149243241Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: [1] Replied this last flush ID: 33619
[2023-10-06T14:33:52.149259652Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: [1] e4a0216e-53b8-4e84-9db0-82a6edb24820 (5eb7460c-1e82-4594-bcda-326a6cf20de3) Active Offline Active ds_transition to Replay
[2023-10-06T14:33:52.149272427Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: [1] Transition from Offline to Replay
[2023-10-06T14:33:52.149285211Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: [1] e4a0216e-53b8-4e84-9db0-82a6edb24820 Transition from Replay to Active
[2023-10-06T14:33:52.149297905Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: [1] e4a0216e-53b8-4e84-9db0-82a6edb24820 (5eb7460c-1e82-4594-bcda-326a6cf20de3) Active Replay Active ds_transition to Active
[2023-10-06T14:33:52.149310338Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: [1] Transition from Replay to Active
[2023-10-06T14:33:52.149323133Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: [1] Starts cmd_loop
[2023-10-06T14:33:52.149336187Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: [1] [fd00:1122:3344:11b::a]:19001 task reports connection:true
[2023-10-06T14:33:52.149348791Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: e4a0216e-53b8-4e84-9db0-82a6edb24820 Active Active Active
[2023-10-06T14:33:52.149361405Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: Set check for repair
[2023-10-06T14:33:53.150142549Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: Checking if live repair is needed
[2023-10-06T14:33:53.150176758Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: No Live Repair required at this time
[2023-10-06T14:33:53.150191439Z]  WARN: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: [1] flow control end 

Right after that is the Stop request:

[2023-10-06T15:08:19.653671524Z]  INFO: propolis-server/vm_controller/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: Requested state Stop via API
[2023-10-06T15:08:19.653726598Z]  INFO: propolis-server/external_request_queue/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: Queuing external request (disposition=Enqueue, request=Stop)
[2023-10-06T15:08:19.653745724Z]  INFO: propolis-server/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: request completed (req_id=52daa4e7-c7cc-4d5f-b17c-6a33f73b63d2, uri=/instance/state, method=PUT, remote_addr=[fd00:1122:3344:11d::1]:38115, local_addr=[fd00:1122:3344:11d::22]:12400, response_code=204)
[2023-10-06T15:08:19.653771563Z]  INFO: propolis-server/vm_state_worker/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: State worker handling event (event=External(Stop))
[2023-10-06T15:08:19.65378553Z]  INFO: propolis-server/vm_state_worker/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: Stopping instance
[2023-10-06T15:08:19.653798454Z]  INFO: propolis-server/vcpu_tasks/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: vCPU paused (vcpu=0)
[2023-10-06T15:08:19.653812411Z]  INFO: propolis-server/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: request completed (req_id=5fb85aba-f7c8-443f-a66f-8782f7bb1372, uri=/instance/state-monitor, method=GET, remote_addr=[fd00:1122:3344:11d::1]:47584, local_addr=[fd00:1122:3344:11d::22]:12400, response_code=200)
[2023-10-06T15:08:19.653831166Z]  INFO: propolis-server/vcpu_tasks/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: vCPU paused (vcpu=1)
[2023-10-06T15:08:19.653845914Z]  INFO: propolis-server/vm_controller/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: Sending pause request to lpc-bhyve-atpic
[2023-10-06T15:08:19.653858628Z]  INFO: propolis-server/vm_controller/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: Sending pause request to lpc-bhyve-atpit
[2023-10-06T15:08:19.653871032Z]  INFO: propolis-server/vm_controller/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: Sending pause request to lpc-bhyve-hpet
[2023-10-06T15:08:19.653883445Z]  INFO: propolis-server/vm_controller/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: Sending pause request to lpc-bhyve-ioapic
[2023-10-06T15:08:19.653895688Z]  INFO: propolis-server/vm_controller/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: Sending pause request to lpc-bhyve-rtc
[2023-10-06T15:08:19.653907872Z]  INFO: propolis-server/vm_controller/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: Sending pause request to bhyve-vcpu-0
[2023-10-06T15:08:19.653919954Z]  INFO: propolis-server/vm_controller/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: Sending pause request to bhyve-vcpu-1
[2023-10-06T15:08:19.653934622Z]  INFO: propolis-server/vm_controller/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: Sending pause request to chipset-i440fx
[2023-10-06T15:08:19.653947326Z]  INFO: propolis-server/vm_controller/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: Sending pause request to pci-piix4-hb
[2023-10-06T15:08:19.65395957Z]  INFO: propolis-server/vm_controller/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: Sending pause request to pci-piix3-lpc
[2023-10-06T15:08:19.653971753Z]  INFO: propolis-server/vm_controller/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: Sending pause request to pci-piix3-pm
[2023-10-06T15:08:19.653984617Z]  INFO: propolis-server/vm_controller/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: Sending pause request to lpc-bhyve-pmtimer
[2023-10-06T15:08:19.653997782Z]  INFO: propolis-server/vm_controller/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: Sending pause request to lpc-uart-com1
[2023-10-06T15:08:19.654009865Z]  INFO: propolis-server/vm_controller/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: Sending pause request to lpc-uart-com3
[2023-10-06T15:08:19.65402305Z]  INFO: propolis-server/vm_controller/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: Sending pause request to lpc-uart-com4
[2023-10-06T15:08:19.654035664Z]  INFO: propolis-server/vm_controller/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: Sending pause request to lpc-uart-com2
[2023-10-06T15:08:19.654047767Z]  INFO: propolis-server/vm_controller/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: Sending pause request to lpc-ps2ctrl
[2023-10-06T15:08:19.65405983Z]  INFO: propolis-server/vm_controller/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: Sending pause request to qemu-lpc-debug
[2023-10-06T15:08:19.654071702Z]  INFO: propolis-server/vm_controller/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: Sending pause request to pci-virtio-viona-0.9.0
[2023-10-06T15:08:19.654083775Z]  INFO: propolis-server/vm_controller/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: Sending pause request to pci-nvme-0.17.0
[2023-10-06T15:08:19.654096048Z]  INFO: propolis-server/vm_controller/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: Sending pause request to block-crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981

Then at the end of the log:

[2023-10-06T17:46:14.074297452Z]  WARN: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: [1] Downstairs not responding, take offline
[2023-10-06T17:46:14.074382373Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: [1] e4a0216e-53b8-4e84-9db0-82a6edb24820 Gone missing, transition from Active to Offline
[2023-10-06T17:46:14.074399676Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: [1] e4a0216e-53b8-4e84-9db0-82a6edb24820 connection to [fd00:1122:3344:11b::a]:19001 closed (looper=1)
[2023-10-06T17:46:14.074416999Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: [1] [fd00:1122:3344:11b::a]:19001 task reports connection:false
[2023-10-06T17:46:14.074431486Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: e4a0216e-53b8-4e84-9db0-82a6edb24820 Active Offline Active
[2023-10-06T17:46:14.074444872Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: [1] [fd00:1122:3344:11b::a]:19001 task reports offline
[2023-10-06T17:46:14.074457966Z]  WARN: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: [1] pm_task rx.recv() is None
[2023-10-06T17:46:15.076184127Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: [1] connecting to [fd00:1122:3344:11b::a]:19001 (looper=1)
Oct 06 17:46:24.173 INFO request completed, response_code: 200, uri: /collect/d8a28397-737e-43fd-9c48-0dbd1626101a, method: GET, req_id: 927560ab-8aea-494a-b446-c2ed8a509ce4, remote_addr: [fd00:1122:3344:10b::3]:52701, local_addr: [fd00:1122:3344:11d::22]:43395, component: dropshot
[2023-10-06T17:46:25.077463593Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
[2023-10-06T17:46:36.079886942Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
[2023-10-06T17:46:47.082491288Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
Oct 06 17:46:54.174 INFO request completed, response_code: 200, uri: /collect/d8a28397-737e-43fd-9c48-0dbd1626101a, method: GET, req_id: dfe349d2-53ae-4646-ab4b-39f73d76fe4e, remote_addr: [fd00:1122:3344:10b::3]:52701, local_addr: [fd00:1122:3344:11d::22]:43395, component: dropshot
[2023-10-06T17:46:58.08496241Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
[2023-10-06T17:47:09.087581302Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
[2023-10-06T17:47:20.090588106Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
Oct 06 17:47:24.176 INFO request completed, response_code: 200, uri: /collect/d8a28397-737e-43fd-9c48-0dbd1626101a, method: GET, req_id: 657b6cab-608f-4d46-98c2-ee39b791cdf2, remote_addr: [fd00:1122:3344:10b::3]:52701, local_addr: [fd00:1122:3344:11d::22]:43395, component: dropshot
[2023-10-06T17:47:31.092572824Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
[2023-10-06T17:47:42.095318256Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
[2023-10-06T17:47:53.098001078Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
Oct 06 17:47:54.177 INFO request completed, response_code: 200, uri: /collect/d8a28397-737e-43fd-9c48-0dbd1626101a, method: GET, req_id: 3fd1a5b5-7f66-4295-a205-7be1224eb2fe, remote_addr: [fd00:1122:3344:10b::3]:52701, local_addr: [fd00:1122:3344:11d::22]:43395, component: dropshot
[2023-10-06T17:48:04.100658873Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
[2023-10-06T17:48:05.101636736Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: [1] connecting to [fd00:1122:3344:11b::a]:19001 (looper=1)
[2023-10-06T17:48:15.103514846Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
Oct 06 17:48:24.179 INFO request completed, response_code: 200, uri: /collect/d8a28397-737e-43fd-9c48-0dbd1626101a, method: GET, req_id: a149db06-24a9-4457-b6d5-121a285adf2d, remote_addr: [fd00:1122:3344:10b::3]:52701, local_addr: [fd00:1122:3344:11d::22]:43395, component: dropshot
[2023-10-06T17:48:26.105427901Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
[2023-10-06T17:48:37.108629278Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
[2023-10-06T17:48:48.110412934Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
Oct 06 17:48:54.180 INFO request completed, response_code: 200, uri: /collect/d8a28397-737e-43fd-9c48-0dbd1626101a, method: GET, req_id: 1170d3fd-333c-4d63-b90c-27a74ef8c15c, remote_addr: [fd00:1122:3344:10b::3]:52701, local_addr: [fd00:1122:3344:11d::22]:43395, component: dropshot
[2023-10-06T17:48:59.113249388Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
[2023-10-06T17:49:10.11650724Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
[2023-10-06T17:49:21.119540376Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
Oct 06 17:49:24.181 INFO request completed, response_code: 200, uri: /collect/d8a28397-737e-43fd-9c48-0dbd1626101a, method: GET, req_id: c179c2e0-a3be-4e8a-9cfe-b33eca72d78e, remote_addr: [fd00:1122:3344:10b::3]:52701, local_addr: [fd00:1122:3344:11d::22]:43395, component: dropshot
[2023-10-06T17:49:32.121818181Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
[2023-10-06T17:49:43.12504948Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
[2023-10-06T17:49:54.127021354Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
Oct 06 17:49:54.182 INFO request completed, response_code: 200, uri: /collect/d8a28397-737e-43fd-9c48-0dbd1626101a, method: GET, req_id: 1cb68c00-4260-4be1-9dc5-3e5b58ce375c, remote_addr: [fd00:1122:3344:10b::3]:52701, local_addr: [fd00:1122:3344:11d::22]:43395, component: dropshot
[2023-10-06T17:49:55.128256214Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: [1] connecting to [fd00:1122:3344:11b::a]:19001 (looper=1)
[2023-10-06T17:50:05.129967338Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
[2023-10-06T17:50:16.132310097Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
Oct 06 17:50:24.184 INFO request completed, response_code: 200, uri: /collect/d8a28397-737e-43fd-9c48-0dbd1626101a, method: GET, req_id: a6308db3-2ea3-47e6-aee2-81826c254f4a, remote_addr: [fd00:1122:3344:10b::3]:52701, local_addr: [fd00:1122:3344:11d::22]:43395, component: dropshot
[2023-10-06T17:50:27.134798853Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
[2023-10-06T17:50:38.137427482Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
[2023-10-06T17:50:49.140171252Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
Oct 06 17:50:54.185 INFO request completed, response_code: 200, uri: /collect/d8a28397-737e-43fd-9c48-0dbd1626101a, method: GET, req_id: 2b88dbe9-ba0f-4142-9dfc-8099b51b8efb, remote_addr: [fd00:1122:3344:10b::3]:52701, local_addr: [fd00:1122:3344:11d::22]:43395, component: dropshot
[2023-10-06T17:51:00.143072865Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
[2023-10-06T17:51:11.144802868Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
[2023-10-06T17:51:22.148118489Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
Oct 06 17:51:24.187 INFO request completed, response_code: 200, uri: /collect/d8a28397-737e-43fd-9c48-0dbd1626101a, method: GET, req_id: 44b01cfb-59a3-4ab6-9694-444f39e7398d, remote_addr: [fd00:1122:3344:10b::3]:52701, local_addr: [fd00:1122:3344:11d::22]:43395, component: dropshot
[2023-10-06T17:51:33.150507751Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
[2023-10-06T17:51:44.153099536Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
[2023-10-06T17:51:45.153946647Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: [1] connecting to [fd00:1122:3344:11b::a]:19001 (looper=1)
Oct 06 17:51:54.188 INFO request completed, response_code: 200, uri: /collect/d8a28397-737e-43fd-9c48-0dbd1626101a, method: GET, req_id: a9088103-320a-4221-948f-cfd4359c15a4, remote_addr: [fd00:1122:3344:10b::3]:52701, local_addr: [fd00:1122:3344:11d::22]:43395, component: dropshot
[2023-10-06T17:51:55.155433897Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
[2023-10-06T17:52:06.158406851Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
[2023-10-06T17:52:17.161128069Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
Oct 06 17:52:24.189 INFO request completed, response_code: 200, uri: /collect/d8a28397-737e-43fd-9c48-0dbd1626101a, method: GET, req_id: 2ea1cbd5-5390-440d-a014-aaf2272a261e, remote_addr: [fd00:1122:3344:10b::3]:52701, local_addr: [fd00:1122:3344:11d::22]:43395, component: dropshot
[2023-10-06T17:52:28.164620175Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
[2023-10-06T17:52:39.166566622Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
[2023-10-06T17:52:50.168941043Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
Oct 06 17:52:54.190 INFO request completed, response_code: 200, uri: /collect/d8a28397-737e-43fd-9c48-0dbd1626101a, method: GET, req_id: 246fd13d-8d36-4f31-ae93-93ffde75d44a, remote_addr: [fd00:1122:3344:10b::3]:52701, local_addr: [fd00:1122:3344:11d::22]:43395, component: dropshot
[2023-10-06T17:53:01.171590302Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
[2023-10-06T17:53:12.174309795Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
[2023-10-06T17:53:23.177319888Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
Oct 06 17:53:24.192 INFO request completed, response_code: 200, uri: /collect/d8a28397-737e-43fd-9c48-0dbd1626101a, method: GET, req_id: 98fd6beb-f9d0-40b5-9c27-71ea208f9cf8, remote_addr: [fd00:1122:3344:10b::3]:52701, local_addr: [fd00:1122:3344:11d::22]:43395, component: dropshot
[2023-10-06T17:53:34.180532457Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
[2023-10-06T17:53:35.181231903Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: [1] connecting to [fd00:1122:3344:11b::a]:19001 (looper=1)
[2023-10-06T17:53:45.183387145Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
Oct 06 17:53:54.193 INFO request completed, response_code: 200, uri: /collect/d8a28397-737e-43fd-9c48-0dbd1626101a, method: GET, req_id: 0aa149e2-cc4c-4051-8e3b-bad649c0b0ea, remote_addr: [fd00:1122:3344:10b::3]:52701, local_addr: [fd00:1122:3344:11d::22]:43395, component: dropshot
[2023-10-06T17:53:56.186178123Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
[2023-10-06T17:54:07.188076342Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
[2023-10-06T17:54:18.190605794Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
Oct 06 17:54:24.194 INFO request completed, response_code: 200, uri: /collect/d8a28397-737e-43fd-9c48-0dbd1626101a, method: GET, req_id: d1399665-41ab-46e7-95cd-b5deef39787d, remote_addr: [fd00:1122:3344:10b::3]:52701, local_addr: [fd00:1122:3344:11d::22]:43395, component: dropshot
[2023-10-06T17:54:29.193184981Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
[2023-10-06T17:54:40.195984537Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
[2023-10-06T17:54:51.198857311Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
Oct 06 17:54:54.196 INFO request completed, response_code: 200, uri: /collect/d8a28397-737e-43fd-9c48-0dbd1626101a, method: GET, req_id: 03552bd0-8152-4d53-9edf-515d37ab8337, remote_addr: [fd00:1122:3344:10b::3]:52701, local_addr: [fd00:1122:3344:11d::22]:43395, component: dropshot
[2023-10-06T17:55:02.201032493Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
[2023-10-06T17:55:13.204048947Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
Oct 06 17:55:24.197 INFO request completed, response_code: 200, uri: /collect/d8a28397-737e-43fd-9c48-0dbd1626101a, method: GET, req_id: d48d4959-8c96-4cb2-bb5e-e664e599d692, remote_addr: [fd00:1122:3344:10b::3]:52701, local_addr: [fd00:1122:3344:11d::22]:43395, component: dropshot
[2023-10-06T17:55:24.205956511Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
[2023-10-06T17:55:25.207944158Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: [1] connecting to [fd00:1122:3344:11b::a]:19001 (looper=1)
[2023-10-06T17:55:35.210120057Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
[2023-10-06T17:55:46.212233406Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
Oct 06 17:55:54.198 INFO request completed, response_code: 200, uri: /collect/d8a28397-737e-43fd-9c48-0dbd1626101a, method: GET, req_id: 9a9defdf-482b-4fb9-9c74-fe404ca009e5, remote_addr: [fd00:1122:3344:10b::3]:52701, local_addr: [fd00:1122:3344:11d::22]:43395, component: dropshot
[2023-10-06T17:55:57.215628267Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
[2023-10-06T17:56:08.218440093Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
[2023-10-06T17:56:19.221615297Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
Oct 06 17:56:24.200 INFO request completed, response_code: 200, uri: /collect/d8a28397-737e-43fd-9c48-0dbd1626101a, method: GET, req_id: 1021d0e5-3fad-429c-be3a-eaf3e853971c, remote_addr: [fd00:1122:3344:10b::3]:52701, local_addr: [fd00:1122:3344:11d::22]:43395, component: dropshot
[2023-10-06T17:56:30.223407133Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
[2023-10-06T17:56:41.226436563Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
[2023-10-06T17:56:52.229185008Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: connect timeout (looper=1)
Oct 06 17:56:54.202 INFO request completed, response_code: 200, uri: /collect/d8a28397-737e-43fd-9c48-0dbd1626101a, method: GET, req_id: f59122c9-77ba-4579-a137-0fe998ac5751, remote_addr: [fd00:1122:3344:10b::3]:52701, local_addr: [fd00:1122:3344:11d::22]:43395, component: dropshot
[2023-10-06T17:57:02.135234676Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: [1] e4a0216e-53b8-4e84-9db0-82a6edb24820 looper connected (looper=1)
[2023-10-06T17:57:02.135302945Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: [1] Proc runs for [fd00:1122:3344:11b::a]:19001 in state Offline
[2023-10-06T17:57:02.135322903Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: [1] client re-new 0 jobs since flush 33619 (=downstairs)
[2023-10-06T17:57:02.135388478Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: [1] upstairs guest_io_ready=TRUE, promote! session 5eb7460c-1e82-4594-bcda-326a6cf20de3
[2023-10-06T17:57:02.135582175Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: [1] downstairs client at [fd00:1122:3344:11b::a]:19001 has UUID 80cfef8a-4350-4b24-8f54-c083b6e8f085
[2023-10-06T17:57:02.135602734Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: [1] Got region def RegionDefinition { block_size: 512, extent_size: Block { value: 131072, shift: 9 }, extent_count: 48, uuid: 80cfef8a-4350-4b24-8f54-c083b6e8f085, encrypted: true, database_read_version: 1, database_write_version: 1 }
[2023-10-06T17:57:02.135617812Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: Returning client:1 UUID:80cfef8a-4350-4b24-8f54-c083b6e8f085 matches
[2023-10-06T17:57:02.135631127Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: [1] send last flush ID to this DS: 33619
[2023-10-06T17:57:02.135644092Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: e4a0216e-53b8-4e84-9db0-82a6edb24820 Active Offline Active
[2023-10-06T17:57:02.135703805Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: [1] Replied this last flush ID: 33619
[2023-10-06T17:57:02.135720667Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: [1] e4a0216e-53b8-4e84-9db0-82a6edb24820 (5eb7460c-1e82-4594-bcda-326a6cf20de3) Active Offline Active ds_transition to Replay
[2023-10-06T17:57:02.135737028Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: [1] Transition from Offline to Replay
[2023-10-06T17:57:02.135752027Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: [1] e4a0216e-53b8-4e84-9db0-82a6edb24820 Transition from Replay to Active
[2023-10-06T17:57:02.135766494Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: [1] e4a0216e-53b8-4e84-9db0-82a6edb24820 (5eb7460c-1e82-4594-bcda-326a6cf20de3) Active Replay Active ds_transition to Active
[2023-10-06T17:57:02.13578027Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: [1] Transition from Replay to Active
[2023-10-06T17:57:02.135793094Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: [1] Starts cmd_loop
[2023-10-06T17:57:02.135806199Z]  INFO: propolis-server/crucible-9db4009b-a3e1-4e1a-ae54-8db32e249981/11571 on oxz_propolis-server_70a26f6a-8e1e-43d5-b85f-bf2525503294: [1] [fd00:1122:3344:11b::a]:19001 task reports connection:true

Start by reading oxidecomputer/propolis#371 and reviewing the archived logs at /net/catacomb/data/staff/augustus/rack3/BRM44220009/logs/propolis-logs.tar.gz, especially the Crucible connection failures before the Stop request. Done means determining whether this report is covered by that issue and, if not, defining a reproducible fix for instance stop behavior.