blockblaz / blockblaz/zeam

Aggregator clock jitter caused by event loop blocking during PQ signature aggregation

Open
#792 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Zig
Stars
97
Forks
39
PR merge metrics
No merged PRs in 30d

Description

## Discovery

Identified via the `lean_tick_interval_duration_seconds` metric and structured clock logs added in #785.

The metric revealed a bimodal distribution on aggregator nodes: a cluster of ticks around 1.1–1.3s paired with compensating short ticks around
0.3–0.5s, while non-aggregator nodes stay consistently near the nominal 0.8s.

Image


Observed in logs:
```
[clock] slot_interval=0 duration=0.814s
[node] received gossip attestation for slot=0 validator=0
[chain] processed gossip attestation for slot=0 validator=0
[node] adding locally produced aggregation to chain: slot=0
[node] published aggregation to network: slot=0
[clock] slot_interval=1 duration=1.290s ← stalled
[clock] slot_interval=2 duration=0.297s ← compensating tick
[clock] slot_interval=3 duration=0.799s ← back to normal
```

## Root cause

PQ signature aggregation and network publishing run synchronously on the xev event loop thread during interval 1. This blocks the loop long enough that the next timer fires late. The clock's catchup loop then immediately advances and fires interval 2 with the remaining shortened time, producing the characteristic long/short pair. The mean stays ~0.8s, hiding the problem from average-based metrics.

## Impact

Attestation aggregation reliably delays the clock on aggregator nodes by ~0.5s per slot. This could affect the timeliness of subsequent interval duties (block proposals, etc.) that depend on the clock firing on schedule.

## Potential fix

Move PQ signature aggregation off the event loop thread — either into a dedicated thread or as an async task — so it does not block timer callbacks. The event loop should only dispatch work, not perform CPU-bound cryptographic operations inline.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing the lean_tick_interval_duration_seconds metric and the structured clock logs added in #785, then follow the xev event-loop path for PQ signature aggregation and network publishing. Confirm that CPU-bound aggregation no longer blocks timer callbacks and that aggregator clock intervals no longer show the long/short jitter pattern.

Written by the indexing model from the issue text.

Assessment

Tech stack
zig
Domain
backend, cryptography, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.