hiero-ledger / hiero-ledger/hiero-consensus-node

reduce block-stream hashing/serialization overhead in BlockStreamManagerImpl

Open
#26,909 0 comments 0 reactions 1 assignee Claimed by @derektriley View on GitHub
Dominant language
Java
Stars
406
Forks
226
Avg merge
3d 4h
Merged PRs (30d)
210

Description

### Background

## Summary

Profiling the block-stream write path (`BlockStreamManagerImpl`'s `ParallelTask`/`SequentialTask` pipeline) turned up three
related inefficiencies in how block items are hashed and written to disk. This issue tracks the fix and the benchmarks used to
validate it.

## Problem

1. **Leaf hashing was serialized unnecessarily.** `SequentialTask.onExecute()` did both the SHA-384 leaf hash (an O(item size)
operation with no dependency on other items) *and* the O(1) Merkle fold-up + writer append in the same strictly-ordered task
chain. Only the fold-up and the write actually need item order — the hash itself could run in parallel, same as serialization
already does.
2. **Every block item was copied 3x when 1 would do.** In the standard file+gRPC config:
`BlockItem.PROTOBUF.toBytes(item).toByteArray()` threw away a zero-copy `Bytes` wrapper to force a full-array copy, and
`FileAndGrpcBlockItemWriter`/`FileBlockItemWriter` copied it again just to hand a `byte[]` to the writer — despite the PBJ
runtime already exposing zero-copy paths (`Bytes.writeTo(MessageDigest)`, `WritableStreamingData.writeBytes(RandomAccessData)`).
3. **`IncrementalStreamingHasher` used a `LinkedList` as a stack**, allocating a `Node` per push and discarding two per
fold, on a hot path that runs once per leaf across up to 5 hashers per block item.

## Proposed Changes

- **`BlockStreamManagerImpl.java`**: moved SHA-384 leaf-hash computation into `ParallelTask` (parallel, per-item), leaving only
`addNodeByHash` (fold-up) and the writer append in `SequentialTask`. Added a `ThreadLocal` for per-thread reuse.
`TRANSACTION_RESULT`'s running-hash update now reuses the same precomputed hash instead of hashing twice.
- **`FileBlockItemWriter.java` / `FileAndGrpcBlockItemWriter.java`**: carry `Bytes` end-to-end instead of unwrapping to
`byte[]`; `writeItem` now calls `writableStreamingData.writeBytes(bytes)` directly (zero-copy).
- **`IncrementalStreamingHasher.java`**: `hashList` changed from `LinkedList` to `Deque` (`ArrayDeque`);
`computeRootHash()` snapshots into a `byte[][]` for the indexed fold since `ArrayDeque` has no `get(int)`.
- Added two JMH benchmarks (`hedera-app/src/jmh/.../blocks/`) to validate and guard against regression:
- `SequentialTaskDesignBenchmark` — current vs. proposed task-split design, with a correctness self-check that both produce
identical Merkle roots.
- `IncrementalStreamingHasherDesignBenchmark` — current `ArrayDeque` vs. a reproduced `LinkedList` implementation, same
self-check.

### Acceptance Criteria

See above

### Dependencies

_No response_

### Definition of Ready (DoR) Checklist

- [ ] Clear acceptance criteria
- [ ] Clear and detailed description
- [ ] Dependencies identified
- [ ] Links to documentation
- [ ] Should be completable in 2-3 Days
- [ ] Initial draft of Low-level design document
- [ ] At least high level test plan
- [ ] Groomed/Estimated

### Definition of Done (DoD) Checklist

- [ ] Acceptance Criteria complete
- [ ] No Codacy issues greater than minor (in new code)
- [ ] JavaDocs updated/created
- [ ] Code commented
- [ ] Unit tests created/updated
- [ ] 80% test code coverage (in new code)
- [ ] Happy Path and major negative cases in HAPI tests as applicable

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.