IntersectMBO / IntersectMBO/cardano-node
[BUG] - The cardano-node crashes when an immutable file is missing
- Dominant language
- Haskell
- Stars
- 3.2k
- Forks
- 754
- Avg merge
- 4d 20h
- Merged PRs (30d)
- 20
Description
**External**
**Area**
*Other*
**Summary**
When a peer requests an older block, and the immutable file containing that block is not accessible on the local file system, the node crashes.
**Steps to reproduce**
Steps to reproduce the behavior:
1. Move the immutable files from their location
2. Wait until such an event occurs (can take a few days)
3. See the node crashing and the messages in the logs
**Expected behavior**
The node should return an error message and continue running.
**System info (please complete the following information):**
- OS Name: Ubuntu]
- OS Version: e.g. 24.04
- Node version: cardano-node 10.5.3 - linux-aarch64 - ghc-9.6 git rev 6c034ec038d8d276a3595e10e2d38643f09bd1f2
- CLI version: cardano-cli 10.11.0.0 - linux-aarch64 - ghc-9.6 git rev 6c034ec038d8d276a3595e10e2d38643f09bd1f2
The OS, node and cli versions are irrelevant here.
**Screenshots and attachments**
This is a log file snippet with such a crash:
```
[relay5:cardano.node.DiffusionInitializationTracer:Critical:7] [2025-12-08 23:38:08.03 UTC] DiffusionErrored ExceptionInHandler 185.19Type = FsResourceDoesNotExist, fsErrorPath = /home/cardano/cardano-node/db/immutable/02258.primary, fsErrorString = "No such file or drom HasCallStack):
prettyCallStack, called at src/System/FS/API/Types.hs:393:23 in fs-api-0.3.0.0-b19f75df8cf0ad4f95eb6d965255bf9a46056a0239d2d1b004e8d
ioToFsError, called at src/System/FS/IO.hs:113:41 in fs-api-0.3.0.0-b19f75df8cf0ad4f95eb6d965255bf9a46056a0239d2d1b004e8d30ef8eba9b2
handleError, called at src/System/FS/IO.hs:109:23 in fs-api-0.3.0.0-b19f75df8cf0ad4f95eb6d965255bf9a46056a0239d2d1b004e8d30ef8eba9b2
rethrowFsError, called at src/System/FS/IO.hs:50:21 in fs-api-0.3.0.0-b19f75df8cf0ad4f95eb6d965255bf9a46056a0239d2d1b004e8d30ef8eba9
hOpen, called at src/System/FS/API.hs:216:43 in fs-api-0.3.0.0-b19f75df8cf0ad4f95eb6d965255bf9a46056a0239d2d1b004e8d30ef8eba9b2:Syst
withFile, called at src/ouroboros-consensus/Ouroboros/Consensus/Storage/ImmutableDB/Impl/Index/Primary.hs:293:5 in ouroboros-consens6eb7c29c52:Ouroboros.Consensus.Storage.ImmutableDB.Impl.Index.Primary
load, called at src/ouroboros-consensus/Ouroboros/Consensus/Storage/ImmutableDB/Impl/Index/Cache.hs:477:21 in ouroboros-consensus-0.29c52:Ouroboros.Consensus.Storage.ImmutableDB.Impl.Index.Cache
readPrimaryIndex, called at src/ouroboros-consensus/Ouroboros/Consensus/Storage/ImmutableDB/Impl/Index/Cache.hs:556:35 in ouroboros-1d659a66eb7c29c52:Ouroboros.Consensus.Storage.ImmutableDB.Impl.Index.Cache
loadPastChunkInfo, called at src/ouroboros-consensus/Ouroboros/Consensus/Storage/ImmutableDB/Impl/Index/Cache.hs:602:26 in ouroborosf1d659a66eb7c29c52:Ouroboros.Consensus.Storage.ImmutableDB.Impl.Index.Cache
getChunkInfo, called at src/ouroboros-consensus/Ouroboros/Consensus/Storage/ImmutableDB/Impl/Index/Cache.hs:671:11 in ouroboros-cons9a66eb7c29c52:Ouroboros.Consensus.Storage.ImmutableDB.Impl.Index.Cache
readOffsets, called at src/ouroboros-consensus/Ouroboros/Consensus/Storage/ImmutableDB/Impl/Index.hs:199:31 in ouroboros-consensus-0c29c52:Ouroboros.Consensus.Storage.ImmutableDB.Impl.Index
readOffsets, called at src/ouroboros-consensus/Ouroboros/Consensus/Storage/ImmutableDB/Impl/Index.hs:53:5 in ouroboros-consensus-0.29c52:Ouroboros.Consensus.Storage.ImmutableDB.Impl.Index
a use of `readOffsets', called at src/ouroboros-consensus/Ouroboros/Consensus/Storage/ImmutableDB/Impl/Index.hs:130:5 in ouroboros-cd659a66eb7c29c52:Ouroboros.Consensus.Storage.ImmutableDB.Impl.Index, fsLimitation = False}))
[relay5:cardano.node.ChainDB:Error:7] [2025-12-08 23:38:08.03 UTC] Immutable DB was already closed. Double closing.
[relay5:cardano.node.ChainDB:Info:7] [2025-12-08 23:38:08.03 UTC] Closed Volatile DB.
[relay5:cardano.node.ChainDB:Info:7] [2025-12-08 23:38:08.03 UTC] Closed db with immutable tip at 5b9fba6656bf5c1f6423d0db814415879118c0ad04086b1a8722e25c240855e033a16603f61d2a2d42265c5fd at slot 173670795
```
**Additional context**
The error occurs when the files in the `db/immutable` folder are not available (for example, another partition is mounted on it - maybe a slower and cheaper storage type, and the partition is not mounted at the time when this error occurs).
Contributor guide
Assessment
This issue has not been assessed yet.