Aggregator clock jitter caused by event loop blocking during PQ signature aggregation
- Dominant language
- Zig
- Stars
- 97
- Forks
- 39
- PR merge metrics
- No merged PRs in 30d
Description
## Discovery
Identified via the `lean_tick_interval_duration_seconds` metric and structured clock logs added in #785.
The metric revealed a bimodal distribution on aggregator nodes: a cluster of ticks around 1.1–1.3s paired with compensating short ticks around
0.3–0.5s, while non-aggregator nodes stay consistently near the nominal 0.8s.
Observed in logs:
```
[clock] slot_interval=0 duration=0.814s
[node] received gossip attestation for slot=0 validator=0
[chain] processed gossip attestation for slot=0 validator=0
[node] adding locally produced aggregation to chain: slot=0
[node] published aggregation to network: slot=0
[clock] slot_interval=1 duration=1.290s ← stalled
[clock] slot_interval=2 duration=0.297s ← compensating tick
[clock] slot_interval=3 duration=0.799s ← back to normal
```
## Root cause
PQ signature aggregation and network publishing run synchronously on the xev event loop thread during interval 1. This blocks the loop long enough that the next timer fires late. The clock's catchup loop then immediately advances and fires interval 2 with the remaining shortened time, producing the characteristic long/short pair. The mean stays ~0.8s, hiding the problem from average-based metrics.
## Impact
Attestation aggregation reliably delays the clock on aggregator nodes by ~0.5s per slot. This could affect the timeliness of subsequent interval duties (block proposals, etc.) that depend on the clock firing on schedule.
## Potential fix
Move PQ signature aggregation off the event loop thread — either into a dedicated thread or as an async task — so it does not block timer callbacks. The event loop should only dispatch work, not perform CPU-bound cryptographic operations inline.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing the lean_tick_interval_duration_seconds metric and the structured clock logs added in #785, then follow the xev event-loop path for PQ signature aggregation and network publishing. Confirm that CPU-bound aggregation no longer blocks timer callbacks and that aggregator clock intervals no longer show the long/short jitter pattern.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- zig
- Domain
- backend, cryptography, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100