electric-sql / electric-sql/electric
Enhancement: Coordinate retry backoff across ShapeStream instances
- Dominant language
- TypeScript
- Stars
- 10.4k
- Forks
- 375
- Avg merge
- 3d 1h
- Merged PRs (30d)
- 18
Description
## Problem
When a proxy in front of Electric fails, each `ShapeStream` instance retries independently with its own backoff state. Even with improved backoff defaults (#3896), 20+ shapes each independently ramping up to 60s max delay means unnecessary aggregate load and slow recovery — if one stream discovers the backend is healthy, the others don't find out until their own next retry fires.
## Proposed solution
Add a shared "connectivity signal" that `ShapeStream` instances can opt into. When any stream gets a successful response, it broadcasts a signal that other streams can listen for to skip their current backoff and retry immediately.
### Sketch
```typescript
// Shared signal — one per app, passed to all ShapeStream instances
const connectivitySignal = new EventTarget()
// In ShapeStreamOptions
interface ShapeStreamOptions {
// ...existing options
connectivitySignal?: EventTarget
}
// On successful response (in #requestShapeLongPoll / #requestShapeSSE):
this.options.connectivitySignal?.dispatchEvent(new Event('connected'))
// During backoff wait:
// Listen for 'connected' event to resolve the backoff delay early
```
This is essentially a **circuit breaker** pattern — the circuit is "open" (backing off) when the proxy is down, and when one stream "closes" it by succeeding, all others benefit immediately.
### Considerations
- **Opt-in**: Only streams that share the same `connectivitySignal` coordinate. Streams targeting different backends should use different signals (or none).
- **Thundering herd mitigation**: When the signal fires, streams should add a small random jitter (e.g., 0-500ms) before retrying, rather than all retrying at the exact same instant.
- **Reset backoff state**: On receiving the signal, streams should reset their backoff counter so they don't immediately jump back to a long delay if the next request also fails.
- **Scope**: This only helps when multiple streams share the same backend/proxy. It's a no-op for single-stream apps.
## Related
- #3896 — Increases default backoff parameters (immediate fix)
- #3895 — Tracks the `onError` retry tight-loop bug
Contributor guide
Assessment
This issue has not been assessed yet.