hiero-ledger / hiero-ledger/hiero-consensus-node
Block Stream Bucket Uploader
- Dominant language
- Java
- Stars
- 406
- Forks
- 226
- Avg merge
- 3d 4h
- Merged PRs (30d)
- 210
Description
To make block streams publicly available as soon as possible, we want to upload them to buckets initially. This requires the support of a new `BlockStreamWriterMode`. In addition to sending blocks via gRPC and writing them to disk, we also must be able to send them to buckets.
**Requirements**
- We need a third `BlockStreamWriterMode`, which sends the blocks to one or more buckets.
- The buckets must be configurable.
- Both the writer mode and the addresses of the buckets must be configurable during runtime (i.e., the node must not be restarted to take the changes into effect).
- If an error occurs, we want to keep a copy of the block on the local disk. It probably makes sense to write the block to a local file initially and upload it once done (instead of writing directly to the bucket).
- We want to upload only one instance of a block. All consensus nodes will try to upload the same block to a bucket, and therefore only the first one can succeed.
- If a consensus node notices the block is already available, it should check if the blocks are the same (via MD5 hash). If they are different, the block should be kept on the local disk, and an alert must be triggered.
- If uploading a block fails, it should be kept on the local disk, and an alert needs to be triggered.
- If the upload was successful, the local copy can be removed after a few hours.
**Questions**
- How should this be configured? Is uploading file `0.0.121` sufficient?
- What is the best way to trigger an alert? An error message in the log?
Contributor guide
Research direction
Start by locating BlockStreamWriterMode and the runtime configuration path, including how file 0.0.121 is configured. Define the bucket upload flow around local copies, duplicate detection by MD5, retention, and alerting. Done means the writer mode and bucket addresses can change at runtime and all stated failure and consistency cases are handled.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- grpc, java
- Domain
- cloud, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100