network: request-ID vs pending-state race in sendBlocksByRangeRequest / ensureBlocksByRootRequest
- Dominant language
- Zig
- Stars
- 97
- Forks
- 39
- PR merge metrics
- No merged PRs in 30d
Description
# network: request-ID-vs-pending-state race in `sendBlocksByRangeRequest` / `ensureBlocksByRootRequest`
Reported by @ch4r10t33r in review of #824 ([review comment](https://github.com/blockblaz/zeam/pull/824#issuecomment-4391952946) — adjust link as needed).
## The bug
In `pkgs/node/src/network.zig`, both `sendBlocksByRangeRequest` (line 661+) and `sendBlocksByRootRequest` (line 607+) share a small but real race window:
```zig
const request_id = self.requestBlocksByRange(peer_id, start_slot, count, handler) catch |err| {
return err;
};
// ← race window: response can arrive here
self.pending_rpc_requests.put(request_id, PendingRPCEntry{ ... }) catch |err| { ... };
```
`requestBlocksByRange`/`requestBlocksByRoot` call the FFI `self.backend.reqresp.sendRequest`, which dispatches the request on the wire and returns the freshly-allocated `request_id` BEFORE the helper inserts the entry into `pending_rpc_requests`.
If a fast peer (or a loopback-style test peer) responds between the FFI return and the `put`, `handleReqRespResponse` runs `snapshotPendingRequest(request_id)` against the not-yet-populated map, gets `null`, and drops the response with the warning:
```
received RPC response for unknown request_id={d}
```
The mock harness today papers over this for tests by buffering responses through `DeferredResponseTask` (see `pkgs/network/src/mock.zig:158` — comment "This fixes the timing issue where responses were delivered before the caller finished setting up request tracking"), but the production race is real and not caught by the mock.
## Severity
`@ch4r10t33r`'s assessment — and I agree:
- This is a **pre-existing bug** in the `blocks_by_root` path (#820 era, not introduced by #824).
- On the **range path** the cost is higher: a single dropped response = a whole batch (up to `MAX_REQUEST_BLOCKS` = 1024) of slots lost, requiring a full status-cycle (`STATUS_INTERVAL_SLOTS` = 8 slots = ~32s on devnet4) to retry. Not a correctness liveness threat but a wasteful catch-up.
## Fix options
### (a) — Reserve the request_id and pre-insert (preferred)
Split the FFI surface so the caller can reserve a `request_id` first, then insert into `pending_rpc_requests`, then dispatch. On dispatch failure, roll back the pending entry. Requires touching `pkgs/network/src/interface.zig::ReqResp` to add a reservation hook AND the Rust `libp2p-glue` side to honor an externally-allocated id (or to expose its internal counter).
### (b) — Dispatch lock + retry-on-miss in response handler
Add a `pending_dispatch_mutex: std.Thread.Mutex` to `ReqRespNetwork`. Hold it across the entire `(FFI sendRequest + put)` pair. In `snapshotPendingRequest`, if the first lookup misses, briefly acquire the same dispatch mutex (with a short timeout, e.g. 50ms) and retry — if a dispatch was in flight, we wait for the put; if it never existed, we still warn.
Pros: contained to `pkgs/node/src/network.zig`, no FFI change.
Cons: lock contention on every response; retry loop with sleep is inelegant.
### (c) — Pre-insert with sentinel; atomic-swap on FFI return
Insert a placeholder entry under a temporary id known to the dispatch thread; have the FFI's response callback recognise the placeholder. Convoluted, not recommended.
**Recommendation: (a).** A single fix lands both `blocks_by_root` and `blocks_by_range`. The FFI change is small (one new function in the trait + a counter exposure on the Rust side).
## Acceptance criteria
- [ ] `sendBlocksByRangeRequest` and `sendBlocksByRootRequest` both have the pending entry in the map BEFORE any FFI dispatch happens.
- [ ] On FFI dispatch failure, the pending entry is removed and any tracked block roots / peer copies are freed.
- [ ] A regression test in `pkgs/network/src/mock.zig` (or a node-level test) that simulates a synchronous-response peer (no `DeferredResponseTask` buffering) and asserts the response is delivered to the handler — i.e., the warning `received RPC response for unknown request_id={d}` is NOT emitted.
- [ ] Existing 11 mock tests + 89 forkchoice/locking tests still green.
## Out of scope
- Per-peer rate limit / DoS handling on the server side — separate follow-up.
- `@min(gap, MAX_REQUEST_BLOCKS)` cycle-gated catch-up cadence — separate follow-up.
cc @ch4r10t33r
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in pkgs/node/src/network.zig at sendBlocksByRangeRequest, sendBlocksByRootRequest, and handleReqRespResponse, then inspect ReqResp in pkgs/network/src/interface.zig and the Rust libp2p-glue side. Run the existing mock and forkchoice/locking tests, and add a synchronous-response regression test in pkgs/network/src/mock.zig. Done means both pending entries exist before dispatch, failures clean them up, and synchronous responses reach their handlers.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, zig
- Domain
- distributed-systems, networking
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100