blockblaz / blockblaz/zeam

perf: req/resp handler holds forkchoice exclusive lock during block processing on libp2p thread

Open
#708 1 comment 0 reactions 0 assignees View on GitHub
performance
Dominant language
Zig
Stars
97
Forks
39
PR merge metrics
No merged PRs in 30d

Description

## Problem

The libp2p rust bridge runs on its own thread (`rustBridgeThread`). `onReqRespResponse` (handling blocks received from peers) fires on that thread and calls directly into `processBlockByRootChunk` → `chain.onBlock()` → `forkChoice.onBlock()` (exclusive `mutex.lock()`) + `forkChoice.updateHead()` (exclusive lock).

Meanwhile the main libxev event loop calls `forkChoice.onInterval()` (also exclusive mutex). During block sync when many blocks are incoming these exclusive locks contend and can stall both sync and the main tick.

**Serving side** (`onReqRespRequest`) is mostly fine:
- `blocks_by_root`: DB-only, no forkchoice lock ✅
- `status`: two brief `lockShared()` calls — OK but blocked if a writer holds the exclusive lock

## Fix

1. **Response handling**: `onReqRespResponse` should not call `chain.onBlock()` directly on the libp2p thread. Received blocks should be queued into a channel/ring buffer and consumed by the main event loop. This is exactly the architecture described in #700 (`xev.Async` dispatch).

2. **Serving status**: Cache an atomically-updated `Status` snapshot (updated in `onBlockFollowup` whenever head/finalized changes). The `status` req/resp handler then serves it lock-free.

3. **General**: Any read from forkchoice for serving req/resp should use a snapshot — take the lock briefly, copy the needed fields, release, then do the work.

## Related

- #700 — long-term fix: `xev.Async` lock-free dispatch (eliminates the need for any lock here)
- PR #695 — introduced `state_mutex` as interim fix for the same class of problem

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with onReqRespResponse, processBlockByRootChunk, onBlockFollowup, and onReqRespRequest, then read issue #700 and PR #695 for the intended dispatch and mutex context. Trace how blocks reach the main libxev event loop and how status is served. Done means response processing no longer holds forkchoice locks on the libp2p thread and status serving uses the described snapshot approach.

Written by the indexing model from the issue text.

Assessment

Tech stack
zig
Domain
distributed-systems, networking, performance
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.