hiero-ledger / hiero-ledger/hiero-consensus-node

Handle "out of disk space" scenario gracefully

Open
#8,205 3 comments 0 reactions 0 assignees View on GitHub
P2 Platform Tech Debt Reduced
Dominant language
Java
Stars
406
Forks
226
Avg merge
3d 4h
Merged PRs (30d)
210

Description

### Problem

This issue can be reproduced with `CryptoBenchMerkleDb.transferPrefetch` in case a node doesn't have enough disk space.
At some point it may fail with rather cryptic exception:
```
2023-08-23 01:18:31.547 Build: 0.42.0-SNAPSHOT (1e497ec)
...
2023-08-23 04:24:22.146 273800000 transactions, TPS (EMA): 18825, TPS (current): 18698

java.lang.IndexOutOfBoundsException
at java.base/java.nio.Buffer.checkIndex(Buffer.java:749)
at java.base/java.nio.HeapByteBuffer.getInt(HeapByteBuffer.java:439)
at com.swirlds.merkledb.files.hashmap.Bucket.getBucketEntryCount(Bucket.java:180)
at com.swirlds.merkledb.files.hashmap.Bucket.findEntryOffset(Bucket.java:377)
at com.swirlds.merkledb.files.hashmap.Bucket.findValue(Bucket.java:230)
at com.swirlds.merkledb.files.hashmap.HalfDiskHashMap.get(HalfDiskHashMap.java:572)
at com.swirlds.merkledb.MerkleDbDataSource.loadLeafRecord(MerkleDbDataSource.java:637)
at com.swirlds.virtualmap.internal.merkle.RecordAccessorImpl.findLeafRecord(RecordAccessorImpl.java:117)
at com.swirlds.virtualmap.internal.merkle.VirtualRootNode.getForModify(VirtualRootNode.java:793)
at com.swirlds.virtualmap.VirtualMap.getForModify(VirtualMap.java:437)
at com.swirlds.benchmark.CryptoBench.transferPrefetch(CryptoBench.java:259)
```
This exception caused by the fact that the following code in `DataFileWriter#moveMmapBuffer` doesn't really do what it's supposed to do in this case:
```
private void moveMmapBuffer(final int currentMmapPos) throws IOException {
try (final FileChannel channel = FileChannel.open(path, StandardOpenOption.READ, StandardOpenOption.WRITE)) {
final MappedByteBuffer newMap =
channel.map(MapMode.READ_WRITE, mmapPositionInFile + currentMmapPos, MMAP_BUF_SIZE);
```
This code is supposed to create a memory buffer that is directly mapped to some disk space. However, this mapping happens asynchronously. Even if there is not enough disk space to be mapped to, `newMap` appears to be normal: you can put values to and get from this buffer. However, it eventually results in a silent mapping error, and then a subsequent to a bucket of the half-disk hash map results in the exception mentioned above.

### Solution

Here is an attempt to validate newly created mmap buffer:
```
/**
* This method addresses the fact that the mapped byte buffer doesn't have the same safeguards
* as the regular FileChannel. It's possible to create a mapped byte buffer even if there is not enough
* disk space to map it. This code is trying to identify this situation and throw an exception.
*
* @param newMap the mapped byte buffer to verify
*/
private static void verifyMmapCorrectness(final MappedByteBuffer newMap) throws IOException {
// theoretically it's possible, we should check it
if (newMap == null) {
throw new IOException("Failed to map file channel to memory");
}
// The idea is to force this mapping to happen. By default, actual mapping happens lazily.
try {
newMap.put(MMAP_BUF_SIZE - 1, (byte) PROBE_VALUE);
if (newMap.get(MMAP_BUF_SIZE - 1) != PROBE_VALUE) {
throw new IOException("Fatal error when creating mmap. Possibly, out of disk memory.");
}
} catch (final Error e) {
throw new IOException(e);
}
}
```

Unfortunately, it didn't work out due to the fact that the system does the actual mapping asynchronously. At this point it's not clear how exactly to validate it, but without it we'll have cryptic errors that are hard to interpret.

### Alternatives

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.