hiero-ledger / hiero-ledger/hiero-consensus-node
reduce block-stream hashing/serialization overhead in BlockStreamManagerImpl
- Dominant language
- Java
- Stars
- 406
- Forks
- 226
- Avg merge
- 3d 4h
- Merged PRs (30d)
- 210
Description
### Background
## Summary
Profiling the block-stream write path (`BlockStreamManagerImpl`'s `ParallelTask`/`SequentialTask` pipeline) turned up three
related inefficiencies in how block items are hashed and written to disk. This issue tracks the fix and the benchmarks used to
validate it.
## Problem
1. **Leaf hashing was serialized unnecessarily.** `SequentialTask.onExecute()` did both the SHA-384 leaf hash (an O(item size)
operation with no dependency on other items) *and* the O(1) Merkle fold-up + writer append in the same strictly-ordered task
chain. Only the fold-up and the write actually need item order — the hash itself could run in parallel, same as serialization
already does.
2. **Every block item was copied 3x when 1 would do.** In the standard file+gRPC config:
`BlockItem.PROTOBUF.toBytes(item).toByteArray()` threw away a zero-copy `Bytes` wrapper to force a full-array copy, and
`FileAndGrpcBlockItemWriter`/`FileBlockItemWriter` copied it again just to hand a `byte[]` to the writer — despite the PBJ
runtime already exposing zero-copy paths (`Bytes.writeTo(MessageDigest)`, `WritableStreamingData.writeBytes(RandomAccessData)`).
3. **`IncrementalStreamingHasher` used a `LinkedList` as a stack**, allocating a `Node` per push and discarding two per
fold, on a hot path that runs once per leaf across up to 5 hashers per block item.
## Proposed Changes
- **`BlockStreamManagerImpl.java`**: moved SHA-384 leaf-hash computation into `ParallelTask` (parallel, per-item), leaving only
`addNodeByHash` (fold-up) and the writer append in `SequentialTask`. Added a `ThreadLocal` for per-thread reuse.
`TRANSACTION_RESULT`'s running-hash update now reuses the same precomputed hash instead of hashing twice.
- **`FileBlockItemWriter.java` / `FileAndGrpcBlockItemWriter.java`**: carry `Bytes` end-to-end instead of unwrapping to
`byte[]`; `writeItem` now calls `writableStreamingData.writeBytes(bytes)` directly (zero-copy).
- **`IncrementalStreamingHasher.java`**: `hashList` changed from `LinkedList` to `Deque` (`ArrayDeque`);
`computeRootHash()` snapshots into a `byte[][]` for the indexed fold since `ArrayDeque` has no `get(int)`.
- Added two JMH benchmarks (`hedera-app/src/jmh/.../blocks/`) to validate and guard against regression:
- `SequentialTaskDesignBenchmark` — current vs. proposed task-split design, with a correctness self-check that both produce
identical Merkle roots.
- `IncrementalStreamingHasherDesignBenchmark` — current `ArrayDeque` vs. a reproduced `LinkedList` implementation, same
self-check.
### Acceptance Criteria
See above
### Dependencies
_No response_
### Definition of Ready (DoR) Checklist
- [ ] Clear acceptance criteria
- [ ] Clear and detailed description
- [ ] Dependencies identified
- [ ] Links to documentation
- [ ] Should be completable in 2-3 Days
- [ ] Initial draft of Low-level design document
- [ ] At least high level test plan
- [ ] Groomed/Estimated
### Definition of Done (DoD) Checklist
- [ ] Acceptance Criteria complete
- [ ] No Codacy issues greater than minor (in new code)
- [ ] JavaDocs updated/created
- [ ] Code commented
- [ ] Unit tests created/updated
- [ ] 80% test code coverage (in new code)
- [ ] Happy Path and major negative cases in HAPI tests as applicable
Contributor guide
Assessment
This issue has not been assessed yet.