hiero-ledger / hiero-ledger/hiero-consensus-node
Halt transaction processing cleanly if the free disk space crosses a threshold
- Dominant language
- Java
- Stars
- 406
- Forks
- 226
- Avg merge
- 3d 4h
- Merged PRs (30d)
- 210
Description
### Problem
It is possible that under some condition (like bad credentials to the AWS/Google buckets) a node is unable to upload record files for an extended period of time. If that were to happen, we need backpressure to prevent the consensus/executor main node from producing blocks that cannot be written to disk, which would cause a loss of record files. If the virtual merkle disk space fills up, the system should also stop writing records and processing transactions.
### Solution
Have a configuration property for the amount of free disk space that must be available for continued operation. If we detect this free disk space threshold has been exceeded, then we should halt handling transactions or participating in gossip or accepting new transactions for gossip until we have more available disk space. Today we will crash messily. It isn't a problem because we can handle a node crashing messily and survive as a network just fine. But we should improve this to degrade cleanly and issue information in the logs (ERROR/FATAL level) indicating that the disk space was used up. We should also have a threshold at which we WARN the node operator of the pending disk full condition. We don't want to actually fill up all disk because if we did, it would make it harder for an operator to recover the system. So we want to monitor this ourselves.
It is probably sufficient to monitor this once per round.
### Alternatives
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.