[Telemetry Data] May 14 ZKSync Sequencer Stall: 5000ms TCP Timeout + Block Production 10x Below Normal (Public Status: Green)
- Dominant language
- No language data
- Stars
- 2.6k
- Forks
- 291
- PR merge metrics
- No merged PRs in 30d
Description
## [Telemetry Data] May 14 ZKSync Sequencer Stall: 5000ms TCP Timeout + Block Production 10x Below Normal (Public Status: Green)
Same monitoring infrastructure that captured the [May 13 Arbitrum stall](https://github.com/flashbots/pm/issues/140) fired twice today on ZKSync Era.
---
### What Phoenix Zero Recorded
**EVENT 1 — 13:57 UTC**
`🟣 SEQUENCER STALL — ZKSYNC`
`RTT: 5000ms | P99: 5000ms | P95: 5000ms`
**EVENT 2 — 14:05 UTC**
`🟣 SEQUENCER STALL — ZKSYNC`
`RTT: 5000ms | P99: 5000ms | P95: 5000ms`
5000ms = hard socket timeout. The sequencer did not respond to TCP packets within 5 seconds. For an HFT router, this is a black hole: transactions submitted in this window are dropped at the transport layer, never reaching a block.
---
### On-Chain Verification — Block Production Rate
Normal ZKSync Era block rate: ~2 seconds/block.
| Block range | Time span | Rate | vs normal |
|-------------|-----------|------|-----------|
| 70098220 → 70098230 | 13:56:25 → 13:57:01 UTC | 3.6s/block | baseline |
| 70098240 → 70098250 | 13:57:26 → 13:58:17 UTC | **5.1s/block** | **2.5x slower** |
| 70098310 → 70098320 | 14:00:57 → 14:02:10 UTC | **7.3s/block** | **3.6x slower** |
| 70098340 → 70098345 | 14:03:22 → 14:04:13 UTC | **10.2s/block** | **5x slower** ← peak |
Block production did not stop entirely — but throughput dropped to 1 tx/block (vs normal 2-4) during the stall window. The RPC layer was returning 5000ms timeouts while the sequencer was producing skeleton blocks at severely degraded rate.
**Public status pages: no incident reported.**
---
### Concurrent Events (Same Window)
| Chain | Time (UTC) | Reading | Type |
|-------|-----------|---------|------|
| Optimism | 13:54–14:04 | P99: 413–574ms sustained | 🔵 SLOW CREEP |
| Base | 14:00 | P99: 2190ms spike | 🔵 SLOW CREEP |
| ZKSync | 13:57, 14:05 | RTT: 5000ms | 🟣 SEQUENCER STALL |
Multi-chain degradation in the same 10-minute window. Base recovered immediately; Optimism sustained degradation for 10 minutes; ZKSync hit the hard timeout ceiling twice.
---
### Context
ZKSync Era's proof system currently has a documented partial liveness vulnerability (source: L2BEAT). Today's 5000ms TCP timeouts are consistent with this ongoing issue manifesting under load.
The block explorer API itself lags ~2.5 hours behind real-time — meaning on-chain verification of the exact stall window requires waiting for indexing. The RTT sensor fires within 2 seconds of the event.
**This is the delta:** public infrastructure reports green. Our transport-layer sensor sees the black hole before the block explorer indexes it.
---
### Infrastructure
Phoenix Zero RTT Oracle probes 4 chains (Base, Arbitrum, Optimism, ZKSync) via `eth_blockNumber` every 2 seconds.
- WebSocket stream: `wss://rtt.phoenix-ai.work/ws`
- Live Telegram alerts: `@Phoenix_Node_Alex_Bot`
- Server: DigitalOcean NYC1 (198.211.103.36)
Previous data point: [May 13 Arbitrum stall — 3081ms, 47s delta vs public RPC](https://github.com/flashbots/pm/issues/140)
Contact: aleksandrkent64@gmail.com
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.