blockblaz / blockblaz/zeam

blocks_by_root response handler is too slow on max-size requests (Hive timeout)

Open
#844 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Zig
Stars
97
Forks
39
PR merge metrics
No merged PRs in 30d

Description

## Summary

The Hive \`reqresp/blocks_by_root/max_request_limit\` test on 2026-05-07 (suite \`1778164266-…\`, image \`993f193\` / v0.4.15) failed against zeam with \`Outbound failure: Timeout\`. Container log shows zeam took **9.3 seconds** between receiving a max-size \`BlocksByRootV1\` request and starting to send the first response chunk — well past the simulator's outbound RPC timeout. The simulator gives up before zeam emits anything on the wire.

## Trace

\`\`\`
13:52:23.036 [reqresp] Received request from 16Uiu2HAm5A9PRitwX99pvF5M9yeiWAEsyfPrC1mKKhE9dBwQQNhc for protocol /leanconsensus/req/blocks_by_root/1/ssz_snappy (1598 bytes)
… 9.3 seconds of normal forkchoice / clock activity, no log line for the request handler …
13:52:32.338 [reqresp] Sent response chunk on channel 1 ← first chunk
13:52:32.338 [reqresp] Sent response chunk on channel 1
… ~1024 chunks within 15 ms …
13:52:32.353 [reqresp] Sent response chunk on channel 1 ← last chunk in this window
\`\`\`

So the actual on-wire send is fast (1024 chunks in 15 ms), but everything before the first chunk takes ~9 s. The simulator times out around the 10 s mark.

## Likely cause (handler + responder shape)

\`pkgs/node/src/node.zig:1180-1207\` is the handler:

\`\`\`zig
.blocks_by_root => |request| {
const roots = request.roots.constSlice();
for (roots) |root| {
if (self.chain.db.loadBlock(database.DbBlocksNamespace, root)) |signed_block_value| {
var signed_block = signed_block_value;
defer signed_block.deinit();
var response = networks.ReqRespResponse{ .blocks_by_root = undefined };
try types.sszClone(self.allocator, types.SignedBlock, signed_block, &response.blocks_by_root);
defer response.deinit();
try responder.sendResponse(&response);
} else {
self.logger.warn(...);
}
}
try responder.finish();
},
\`\`\`

Two suspect properties:

1. **Per-root work is sequential**: 1024 \`db.loadBlock\` calls, 1024 \`sszClone\` (which is serialize+deserialize — every block is allocated and round-tripped), 1024 FFI hops into the rust bridge, 1024 SSZ-encode-and-frame calls on the rust side. None of this is parallel, none of it is batched.
2. **Chunks are sent one by one but only land on the wire in a single 15 ms burst** (see the trace — every \`Sent response chunk\` line is at \`32.338\` or \`32.353\`). That timing pattern is consistent with the rust bridge accumulating \`SwarmCommand::SendRpcResponseChunk\` items in the per-network channel and the QUIC stream only flushing once the responder calls \`finish()\`. Worth confirming with a tcpdump on the loopback during a local rerun.

If (2) is real, then even on a fast handler the simulator wouldn't see anything until \`finish()\` lands, which means the timeout is structural and not just a function of how many roots the request has. A 1-block request would have the same shape, just with a smaller pre-finish wait.

## Reproduction

\`\`\`bash
# locally
cd hive && ./hive --client zeam --sim lean --sim.limit 'reqresp/blocks_by_root/max_request_limit'

# look for the 9-second gap in /tmp/.../zeam_devnet4/client-….log between
# "[reqresp] Received request from … for protocol /…/blocks_by_root/1/ssz_snappy (… bytes)"
# "[reqresp] Sent response chunk on channel 1"
\`\`\`

## Suggested investigation order

1. Add timestamps around \`db.loadBlock\` and \`sszClone\` in the handler to see where the 9 s actually goes (DB cold cache? sszClone allocator pressure? FFI hop?).
2. Check whether the rust bridge actually flushes per-chunk or batches at \`finish()\` — \`grep SendRpcResponseChunk rust/libp2p-glue/src/lib.rs\` for the event-loop arm and look at when \`reqresp.send_response_chunk\` actually hits the swarm.
3. If the responder is genuinely flushing per chunk, profile \`sszClone\` for \`SignedBlock\` — that's likely the long pole.
4. Consider deduplicating roots in the handler before doing any work: the simulator's max-size payload is 1024 copies of the same root.

## Hive evidence

- Suite: https://hive.leanroadmap.org/suite.html?suiteid=1778164266-ea7641a3b8c224ad3bd80e36b6b8cc77.json&suitename=reqresp&client=zeam_devnet4
- Test: \`reqresp/blocks_by_root/max_request_limit\`
- Image: \`zeam_devnet4: 993f193\` (v0.4.15)
- Simulator error: \`client should accept max-size request: Outbound failure: Timeout\`

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce with the Hive command in the issue, then trace pkgs/node/src/node.zig:1180-1207 around db.loadBlock, sszClone, and responder.sendResponse. Inspect SendRpcResponseChunk handling in rust/libp2p-glue/src/lib.rs and use timestamps or tcpdump to determine whether preparation or flushing causes the delay. Done means the pre-first-chunk bottleneck and response-flush behavior are measured and the max-size test no longer times out.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, zig
Domain
networking, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.