hiero-ledger / hiero-ledger/hiero-consensus-node

Block Stream Bucket Uploader

Open
#16,768 0 comments 0 reactions 0 assignees View on GitHub
Epic
Dominant language
Java
Stars
406
Forks
226
Avg merge
3d 4h
Merged PRs (30d)
210

Description

To make block streams publicly available as soon as possible, we want to upload them to buckets initially. This requires the support of a new `BlockStreamWriterMode`. In addition to sending blocks via gRPC and writing them to disk, we also must be able to send them to buckets.

**Requirements**
- We need a third `BlockStreamWriterMode`, which sends the blocks to one or more buckets.
- The buckets must be configurable.
- Both the writer mode and the addresses of the buckets must be configurable during runtime (i.e., the node must not be restarted to take the changes into effect).
- If an error occurs, we want to keep a copy of the block on the local disk. It probably makes sense to write the block to a local file initially and upload it once done (instead of writing directly to the bucket).
- We want to upload only one instance of a block. All consensus nodes will try to upload the same block to a bucket, and therefore only the first one can succeed.
- If a consensus node notices the block is already available, it should check if the blocks are the same (via MD5 hash). If they are different, the block should be kept on the local disk, and an alert must be triggered.
- If uploading a block fails, it should be kept on the local disk, and an alert needs to be triggered.
- If the upload was successful, the local copy can be removed after a few hours.

**Questions**
- How should this be configured? Is uploading file `0.0.121` sufficient?
- What is the best way to trigger an alert? An error message in the log?

Contributor guide

Open the contributing guide

Research direction

Start by locating BlockStreamWriterMode and the runtime configuration path, including how file 0.0.121 is configured. Define the bucket upload flow around local copies, duplicate detection by MD5, retention, and alerting. Done means the writer mode and bucket addresses can change at runtime and all stated failure and consistency cases are handled.

Written by the indexing model from the issue text.

Assessment

Tech stack
grpc, java
Domain
cloud, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.