Ungraceful shutdown near a GRANDPA authority set change permanently corrupts the ledger arena (v1.0.1)

Start with sc_consensus_grandpa::import::GrandpaBlockImport::import_justification in grandpa/src/import.rs:859 and storage-core/src/arena.rs:1662, then trace the ledger_8 post_block_update path shown in the WASM backtrace. No repository test or target file is named; completion requires a reproduced durability failure and a maintainer-approved fix or verified recovery and shutdown guidance.

Summary

On mainnet, an ungraceful shutdown of midnight-node v1.0.1-5edf8ddd left the node permanently unable to import blocks. Recovery required wiping the Midnight database and resyncing.

The node was killed mid-operation by a Kubernetes pod eviction. On the way down it logged Dropped HF storage after rollback. It restarted cleanly and rejoined the network. Nine minutes later it imported block #2,098,609, which carried a GRANDPA authority set change, and panicked inside import_justification on an assertion the code treats as unreachable. That panic happened partway through the ledger write for the block, leaving a ParityDB arena key unpersisted. Every restart since fails on the same missing key.

Environment

• midnight-node v1.0.1-5edf8ddd
• Mainnet
• Kubernetes (EKS, eu-central-1)
• Co-located Cardano node and cardano-db-sync

Sequence

1. Ungraceful shutdown (02:21:14 UTC)

Pod evicted by the cluster autoscaler. Node was healthy immediately prior, at 19 peers, Imported #2098525.

2026-08-12 02:21:14 Essential task `txpool-background` failed. Shutting down service.
2026-08-12 02:21:15 Dropped HF storage after rollback

2. Restart, looks healthy (02:24:38 to 02:28:19 UTC)

Node restarts, rejoins, climbs back to 19 peers.

3. GRANDPA panic on the authority set change block (02:30:49 UTC)

Why this happened: the rollback during the shutdown left the node's record of the GRANDPA validator set out of step with the chain. When this block asked it to change that set, the node compared what it had on disk against what the block expected, found they did not match, and hit a check that assumes such a mismatch is impossible. Rather than continue, it panicked.

2026-08-12 02:30:49 👴 Applying authority set change scheduled at block #2098609
2026-08-12 02:30:52 👴 Imported justification for block #2098609 that triggers command Changing authorities, signaling voter.

Version: 1.0.1-5edf8ddd

Thread 'tokio-rt-worker' panicked at 'returns Ok when no authority set change should be enacted; qed;',
  /root/.cargo/git/checkouts/polkadot-sdk-dee0edd6eefa0594/2e4dd0b/substrate/client/consensus/grandpa/src/import.rs:859

   0: sp_panic_handler::set::{{closure}}
   1: <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call
      std::panicking::panic_with_hook
   2: std::panicking::panic_handler::{{closure}}
   3: std::sys::backtrace::__rust_end_short_backtrace
   4: __rustc::rust_begin_unwind
   5: core::panicking::panic_fmt
   6: sc_consensus_grandpa::import::GrandpaBlockImport<BE,Block,Client,SC>::import_justification
   7: <sc_consensus_grandpa::import::GrandpaBlockImport<BE,Block,Client,SC> as sc_consensus::block_import::BlockImport<Block>>::import_block::{{closure}}
   8: <alloc::boxed::Box<dyn sc_consensus::block_import::BlockImport<B>+…> as sc_consensus::block_import::BlockImport<B>>::import_block::{{closure}}
   9: futures_util::future::future::FutureExt::poll_unpin
  10: sc_consensus::import_queue::basic_queue::BlockImportWorker<B>::new::{{closure}}

This is a bug. Please report it at:

    https://github.com/midnightntwrk/midnight-node/issues

Block #2,098,609 is the last block the node ever imported.

4. Deterministic arena failure on every retry (02:30:54 UTC onward)

Why this happened: the panic in step 3 landed partway through writing this block's ledger data to disk. Some of it was saved and some was not. On every restart the node tries the same block again, needs one of the pieces that never got written, cannot find it, and fails the same way. Nothing about restarting changes that, so it fails identically every time.

WASM backtrace:
error while executing at wasm backtrace:
    0:  0x2c229 - midnight_node_runtime.wasm!midnight_node_ledger::host_api::ledger_8::ledger_8_bridge::extern_host_function_impls::post_block_update::h4d7a8cc0de28416e
    1: 0x18add6 - midnight_node_runtime.wasm!<… as frame_support::traits::hooks::OnFinalize<BlockNumber>>::on_finalize::h90153380800f6285
    2:  0xc0c1b - midnight_node_runtime.wasm!frame_executive::Executive<…>::execute_block::h22acd8cc3e361db7
    3:  0xc060c - midnight_node_runtime.wasm!Core_execute_block

Thread 'tokio-rt-worker' panicked at 'root should be in the arena (T=midnight_node_ledger::ledger_8::common::api::ledger::Ledger<midnight_storage_core::db::paritydb::ParityDb>): Custom { kind: NotFound, error: "BackendLoader::get(): key 6ae5d21acac19ffd8f05598bdfccbca2061e7a79e2ce81076e6f9377256e643f not in storage arena. Are you sure you persisted this key or one of its ancestors?" }',
  /root/.cargo/git/checkouts/shadow-midnight-ledger-7d9ec0c51f3cacce/397da47/storage-core/src/arena.rs:1662

2026-08-12 02:30:54 💔 Error importing block 0x786029ce277c8956d04f57ed2d4c99c2875f107dd8ed8b2151c9c612a50e2f3e: consensus error: Import failed: Import failed: Error at calling runtime api: Execution failed: Execution aborted due to trap: host code panicked while being called by the runtime: root should be in the arena … key 6ae5d21a… not in storage arena.

The same key 6ae5d21acac19ffd8f05598bdfccbca2061e7a79e2ce81076e6f9377256e643f is missing on every attempt. The node stays alive but stuck, and its peer count falls to zero:

2026-08-12 05:28:20 💤 Idle (0 peers), best: #2098609 (0xbc90…1ff0), finalized #2098609 (0xbc90…1ff0)

Impact

Roughly three hours of lost block production on a mainnet validator. Mitigated by failing over to a standby. The affected node could only be recovered by wiping its Midnight volume and resyncing. The Cardano volumes were unaffected.

Questions

1. Should an ungraceful shutdown near a GRANDPA authority set change be able to leave the ledger DB permanently unbootable, or is this a durability bug?
2. Is there a recovery path short of wiping the Midnight volume, for example rolling the ledger back to the last consistent block?
3. Is a fix planned for the v1.0.x line?
4. Is there a documented safe shutdown procedure, for instance a specific signal or a grace period the node needs to reach a consistent on-disk state?
5. Is the authority set change block a necessary ingredient here, or can any ungraceful shutdown produce this?

Note on a possibly unrelated log line

About a minute before the panic the node logged a Cardano stability error while its Cardano view was catching up after the restart:

2026-08-12 02:29:48 Get stable block by hash failed: Block with hash 355634cefa75a687efbc1cab6e1c062384b3b8cbbe37314d02cd56d3e94575eb is not stable yet: block 13795440 requires latest block >= 13797600, but latest block is 13797599.
2026-08-12 02:29:48 Failed to create mc_hash inherent data for verification: Main chain state 355634cefa75a687efbc1cab6e1c062384b3b8cbbe37314d02cd56d3e94575eb referenced in imported block at slot 297750298 with timestamp 1786501788000 not found
2026-08-12 02:29:48 💔 Verification failed for block 0x4686c7eb51f98cd1043e9b890bf48bcc1af59bc0db746568737521a01ce9da61 received from (12D3KooWDavLZDsAvdD7oDP6izt7U3iMzNXcmEFCzzXDbYSaPCSV): "Main chain state 355634ce… not found"

Our reading of this:

• A peer sent a Midnight block referencing Cardano block 355634ce… at height 13,795,440. The stability rule requires the Cardano tip to be at least 13,795,440 + 2,160 = 13,797,600 before that reference counts as stable. Our Cardano view was at 13,797,599, one block short.
• Because the reference was not stable, the node could not build the mc_hash inherent data needed to verify the block, so the main chain state came back as not found.
• It therefore rejected the peer's Midnight block 0x4686c7eb… as unverifiable.
• This was most likely transient, since our Cardano view was still catching up after the restart and only needed one more block.

We do not think this is connected to the panic that followed, and are including it only for completeness.