Region extent db file corruption after sled was rebooted with kmdb enabled

We have several downstairs on rack3 that hit this kind of error after the crucible zone came back from a sled reboot:

[ Oct  7 22:13:57 Disabled. ]
[ Oct  7 22:13:57 Enabled. ]
[ Oct  7 22:13:57 Executing start method ("/opt/oxide/lib/svc/manifest/crucible/downstairs.sh"). ]
{"msg":"current number of open files limit 65536 is already the maximum","v":0,"name":"crucible","level":30,"time":"2023-10-07T22:13:57.390738293Z","hostname":"oxz_crucible_3b56e56b-8328-45c1-b459-1f14bbea5e36","pid":8992}
{"msg":"Opened existing region file \"/data/regions/0a7868bd-9778-49c5-bc05-45892a9f7ec6/region.json\"","v":0,"name":"crucible","level":30,"time":"2023-10-07T22:13:57.391071316Z","hostname":"oxz_crucible_3b56e56b-8328-45c1-b459-1f14bbea5e36","pid":8992}
{"msg":"Database read version 1","v":0,"name":"crucible","level":30,"time":"2023-10-07T22:13:57.391101694Z","hostname":"oxz_crucible_3b56e56b-8328-45c1-b459-1f14bbea5e36","pid":8992}
{"msg":"Database write version 1","v":0,"name":"crucible","level":30,"time":"2023-10-07T22:13:57.39111552Z","hostname":"oxz_crucible_3b56e56b-8328-45c1-b459-1f14bbea5e36","pid":8992}
{"msg":"Error: Open of db file \"/data/regions/0a7868bd-9778-49c5-bc05-45892a9f7ec6/00/000/028.db\" for extent#40 returned: file is not a database","v":0,"name":"crucible","level":50,"time":"2023-10-07T22:13:57.455828553Z","hostname":"oxz_crucible_3b56e56b-8328-45c1-b459-1f14bbea5e36","pid":8992}
Error: Open of db file "/data/regions/0a7868bd-9778-49c5-bc05-45892a9f7ec6/00/000/028.db" for extent#40 returned: file is not a database
[ Oct  7 22:13:57 Stopping because service exited with an error. ]

The sled in question (sled 10 on rack3) was rebooted with kmdb enabled to troubleshoot https://github.com/oxidecomputer/stlouis/issues/281. Three other regions also ran into the same kind of error when their crucible zones came back up after the sled reboot.

For now, I have disabled these downstairs to prevent the indefinite retries to bring up the SMF service:

root@oxz_crucible_c5d44f20-3fd7-4859-9386-e81239dc413e:~# svcadm disable downstairs-a101225c-5829-45d3-9129-792a6b13fd06

root@oxz_crucible_3b56e56b-8328-45c1-b459-1f14bbea5e36:~# svcadm disable downstairs-0a7868bd-9778-49c5-bc05-45892a9f7ec6

root@oxz_crucible_3b56e56b-8328-45c1-b459-1f14bbea5e36:~# svcadm disable downstairs-612d3da0-8332-4483-b2a8-04e4e013dccd

root@oxz_crucible_864fdb33-a36d-4bb0-96e6-a3b3b057ba8b:~# svcadm disable downstairs-14f17cc8-4857-49c4-861c-a1bb831a4dbb

The complete crucible zone logs are located in catacomb:/staff/augustus/rack3/instances/7f82a998-6f4f-4e53-bfde-325f339a850c ( pool/ext/*/crypt/debug/oxz_crucible_3b56e56b-8328-45c1-b459-1f14bbea5e36/oxide-crucible-downstairs:downstairs-0a7868bd-9778-49c5-bc05-45892a9f7ec6.log.*).

The region tar file is being exported from the rack. I'll add the link if we decide to upload it to catacomb for future reference..

Start by reviewing the complete crucible zone logs at catacomb:/staff/augustus/rack3/instances/7f82a998-6f4f-4e53-bfde-325f339a850c and the exported region tar file. Compare the affected region extent database files and reboot context to determine the corruption cause; done means the failure is explained and a corrective change or recovery path is identified.