hiero-ledger / hiero-ledger/hiero-consensus-node
Handle "out of disk space" scenario gracefully
- Dominant language
- Java
- Stars
- 406
- Forks
- 226
- Avg merge
- 3d 4h
- Merged PRs (30d)
- 210
Description
### Problem
This issue can be reproduced with `CryptoBenchMerkleDb.transferPrefetch` in case a node doesn't have enough disk space.
At some point it may fail with rather cryptic exception:
```
2023-08-23 01:18:31.547 Build: 0.42.0-SNAPSHOT (1e497ec)
...
2023-08-23 04:24:22.146 273800000 transactions, TPS (EMA): 18825, TPS (current): 18698
java.lang.IndexOutOfBoundsException
at java.base/java.nio.Buffer.checkIndex(Buffer.java:749)
at java.base/java.nio.HeapByteBuffer.getInt(HeapByteBuffer.java:439)
at com.swirlds.merkledb.files.hashmap.Bucket.getBucketEntryCount(Bucket.java:180)
at com.swirlds.merkledb.files.hashmap.Bucket.findEntryOffset(Bucket.java:377)
at com.swirlds.merkledb.files.hashmap.Bucket.findValue(Bucket.java:230)
at com.swirlds.merkledb.files.hashmap.HalfDiskHashMap.get(HalfDiskHashMap.java:572)
at com.swirlds.merkledb.MerkleDbDataSource.loadLeafRecord(MerkleDbDataSource.java:637)
at com.swirlds.virtualmap.internal.merkle.RecordAccessorImpl.findLeafRecord(RecordAccessorImpl.java:117)
at com.swirlds.virtualmap.internal.merkle.VirtualRootNode.getForModify(VirtualRootNode.java:793)
at com.swirlds.virtualmap.VirtualMap.getForModify(VirtualMap.java:437)
at com.swirlds.benchmark.CryptoBench.transferPrefetch(CryptoBench.java:259)
```
This exception caused by the fact that the following code in `DataFileWriter#moveMmapBuffer` doesn't really do what it's supposed to do in this case:
```
private void moveMmapBuffer(final int currentMmapPos) throws IOException {
try (final FileChannel channel = FileChannel.open(path, StandardOpenOption.READ, StandardOpenOption.WRITE)) {
final MappedByteBuffer newMap =
channel.map(MapMode.READ_WRITE, mmapPositionInFile + currentMmapPos, MMAP_BUF_SIZE);
```
This code is supposed to create a memory buffer that is directly mapped to some disk space. However, this mapping happens asynchronously. Even if there is not enough disk space to be mapped to, `newMap` appears to be normal: you can put values to and get from this buffer. However, it eventually results in a silent mapping error, and then a subsequent to a bucket of the half-disk hash map results in the exception mentioned above.
### Solution
Here is an attempt to validate newly created mmap buffer:
```
/**
* This method addresses the fact that the mapped byte buffer doesn't have the same safeguards
* as the regular FileChannel. It's possible to create a mapped byte buffer even if there is not enough
* disk space to map it. This code is trying to identify this situation and throw an exception.
*
* @param newMap the mapped byte buffer to verify
*/
private static void verifyMmapCorrectness(final MappedByteBuffer newMap) throws IOException {
// theoretically it's possible, we should check it
if (newMap == null) {
throw new IOException("Failed to map file channel to memory");
}
// The idea is to force this mapping to happen. By default, actual mapping happens lazily.
try {
newMap.put(MMAP_BUF_SIZE - 1, (byte) PROBE_VALUE);
if (newMap.get(MMAP_BUF_SIZE - 1) != PROBE_VALUE) {
throw new IOException("Fatal error when creating mmap. Possibly, out of disk memory.");
}
} catch (final Error e) {
throw new IOException(e);
}
}
```
Unfortunately, it didn't work out due to the fact that the system does the actual mapping asynchronously. At this point it's not clear how exactly to validate it, but without it we'll have cryptic errors that are hard to interpret.
### Alternatives
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.