blockblaz / blockblaz/zeam

network: request-ID vs pending-state race in sendBlocksByRangeRequest / ensureBlocksByRootRequest

Open
#833 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Zig
Stars
97
Forks
39
PR merge metrics
No merged PRs in 30d

Description

# network: request-ID-vs-pending-state race in `sendBlocksByRangeRequest` / `ensureBlocksByRootRequest`

Reported by @ch4r10t33r in review of #824 ([review comment](https://github.com/blockblaz/zeam/pull/824#issuecomment-4391952946) — adjust link as needed).

## The bug

In `pkgs/node/src/network.zig`, both `sendBlocksByRangeRequest` (line 661+) and `sendBlocksByRootRequest` (line 607+) share a small but real race window:

```zig
const request_id = self.requestBlocksByRange(peer_id, start_slot, count, handler) catch |err| {
return err;
};
// ← race window: response can arrive here

self.pending_rpc_requests.put(request_id, PendingRPCEntry{ ... }) catch |err| { ... };
```

`requestBlocksByRange`/`requestBlocksByRoot` call the FFI `self.backend.reqresp.sendRequest`, which dispatches the request on the wire and returns the freshly-allocated `request_id` BEFORE the helper inserts the entry into `pending_rpc_requests`.

If a fast peer (or a loopback-style test peer) responds between the FFI return and the `put`, `handleReqRespResponse` runs `snapshotPendingRequest(request_id)` against the not-yet-populated map, gets `null`, and drops the response with the warning:

```
received RPC response for unknown request_id={d}
```

The mock harness today papers over this for tests by buffering responses through `DeferredResponseTask` (see `pkgs/network/src/mock.zig:158` — comment "This fixes the timing issue where responses were delivered before the caller finished setting up request tracking"), but the production race is real and not caught by the mock.

## Severity

`@ch4r10t33r`'s assessment — and I agree:

- This is a **pre-existing bug** in the `blocks_by_root` path (#820 era, not introduced by #824).
- On the **range path** the cost is higher: a single dropped response = a whole batch (up to `MAX_REQUEST_BLOCKS` = 1024) of slots lost, requiring a full status-cycle (`STATUS_INTERVAL_SLOTS` = 8 slots = ~32s on devnet4) to retry. Not a correctness liveness threat but a wasteful catch-up.

## Fix options

### (a) — Reserve the request_id and pre-insert (preferred)

Split the FFI surface so the caller can reserve a `request_id` first, then insert into `pending_rpc_requests`, then dispatch. On dispatch failure, roll back the pending entry. Requires touching `pkgs/network/src/interface.zig::ReqResp` to add a reservation hook AND the Rust `libp2p-glue` side to honor an externally-allocated id (or to expose its internal counter).

### (b) — Dispatch lock + retry-on-miss in response handler

Add a `pending_dispatch_mutex: std.Thread.Mutex` to `ReqRespNetwork`. Hold it across the entire `(FFI sendRequest + put)` pair. In `snapshotPendingRequest`, if the first lookup misses, briefly acquire the same dispatch mutex (with a short timeout, e.g. 50ms) and retry — if a dispatch was in flight, we wait for the put; if it never existed, we still warn.

Pros: contained to `pkgs/node/src/network.zig`, no FFI change.
Cons: lock contention on every response; retry loop with sleep is inelegant.

### (c) — Pre-insert with sentinel; atomic-swap on FFI return

Insert a placeholder entry under a temporary id known to the dispatch thread; have the FFI's response callback recognise the placeholder. Convoluted, not recommended.

**Recommendation: (a).** A single fix lands both `blocks_by_root` and `blocks_by_range`. The FFI change is small (one new function in the trait + a counter exposure on the Rust side).

## Acceptance criteria

- [ ] `sendBlocksByRangeRequest` and `sendBlocksByRootRequest` both have the pending entry in the map BEFORE any FFI dispatch happens.
- [ ] On FFI dispatch failure, the pending entry is removed and any tracked block roots / peer copies are freed.
- [ ] A regression test in `pkgs/network/src/mock.zig` (or a node-level test) that simulates a synchronous-response peer (no `DeferredResponseTask` buffering) and asserts the response is delivered to the handler — i.e., the warning `received RPC response for unknown request_id={d}` is NOT emitted.
- [ ] Existing 11 mock tests + 89 forkchoice/locking tests still green.

## Out of scope

- Per-peer rate limit / DoS handling on the server side — separate follow-up.
- `@min(gap, MAX_REQUEST_BLOCKS)` cycle-gated catch-up cadence — separate follow-up.

cc @ch4r10t33r

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in pkgs/node/src/network.zig at sendBlocksByRangeRequest, sendBlocksByRootRequest, and handleReqRespResponse, then inspect ReqResp in pkgs/network/src/interface.zig and the Rust libp2p-glue side. Run the existing mock and forkchoice/locking tests, and add a synchronous-response regression test in pkgs/network/src/mock.zig. Done means both pending entries exist before dispatch, failures clean them up, and synchronous responses reach their handlers.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, zig
Domain
distributed-systems, networking
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.