flashbots / flashbots/pm

[Telemetry Data] May 14 ZKSync Sequencer Stall: 5000ms TCP Timeout + Block Production 10x Below Normal (Public Status: Green)

Open
#141 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
No language data
Stars
2.6k
Forks
291
PR merge metrics
No merged PRs in 30d

Description

## [Telemetry Data] May 14 ZKSync Sequencer Stall: 5000ms TCP Timeout + Block Production 10x Below Normal (Public Status: Green)

Same monitoring infrastructure that captured the [May 13 Arbitrum stall](https://github.com/flashbots/pm/issues/140) fired twice today on ZKSync Era.

---

### What Phoenix Zero Recorded

**EVENT 1 — 13:57 UTC**
`🟣 SEQUENCER STALL — ZKSYNC`
`RTT: 5000ms | P99: 5000ms | P95: 5000ms`

**EVENT 2 — 14:05 UTC**
`🟣 SEQUENCER STALL — ZKSYNC`
`RTT: 5000ms | P99: 5000ms | P95: 5000ms`

5000ms = hard socket timeout. The sequencer did not respond to TCP packets within 5 seconds. For an HFT router, this is a black hole: transactions submitted in this window are dropped at the transport layer, never reaching a block.

---

### On-Chain Verification — Block Production Rate

Normal ZKSync Era block rate: ~2 seconds/block.

| Block range | Time span | Rate | vs normal |
|-------------|-----------|------|-----------|
| 70098220 → 70098230 | 13:56:25 → 13:57:01 UTC | 3.6s/block | baseline |
| 70098240 → 70098250 | 13:57:26 → 13:58:17 UTC | **5.1s/block** | **2.5x slower** |
| 70098310 → 70098320 | 14:00:57 → 14:02:10 UTC | **7.3s/block** | **3.6x slower** |
| 70098340 → 70098345 | 14:03:22 → 14:04:13 UTC | **10.2s/block** | **5x slower** ← peak |

Block production did not stop entirely — but throughput dropped to 1 tx/block (vs normal 2-4) during the stall window. The RPC layer was returning 5000ms timeouts while the sequencer was producing skeleton blocks at severely degraded rate.

**Public status pages: no incident reported.**

---

### Concurrent Events (Same Window)

| Chain | Time (UTC) | Reading | Type |
|-------|-----------|---------|------|
| Optimism | 13:54–14:04 | P99: 413–574ms sustained | 🔵 SLOW CREEP |
| Base | 14:00 | P99: 2190ms spike | 🔵 SLOW CREEP |
| ZKSync | 13:57, 14:05 | RTT: 5000ms | 🟣 SEQUENCER STALL |

Multi-chain degradation in the same 10-minute window. Base recovered immediately; Optimism sustained degradation for 10 minutes; ZKSync hit the hard timeout ceiling twice.

---

### Context

ZKSync Era's proof system currently has a documented partial liveness vulnerability (source: L2BEAT). Today's 5000ms TCP timeouts are consistent with this ongoing issue manifesting under load.

The block explorer API itself lags ~2.5 hours behind real-time — meaning on-chain verification of the exact stall window requires waiting for indexing. The RTT sensor fires within 2 seconds of the event.

**This is the delta:** public infrastructure reports green. Our transport-layer sensor sees the black hole before the block explorer indexes it.

---

### Infrastructure

Phoenix Zero RTT Oracle probes 4 chains (Base, Arbitrum, Optimism, ZKSync) via `eth_blockNumber` every 2 seconds.

- WebSocket stream: `wss://rtt.phoenix-ai.work/ws`
- Live Telegram alerts: `@Phoenix_Node_Alex_Bot`
- Server: DigitalOcean NYC1 (198.211.103.36)

Previous data point: [May 13 Arbitrum stall — 3081ms, 47s delta vs public RPC](https://github.com/flashbots/pm/issues/140)

Contact: aleksandrkent64@gmail.com

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.