electric-sql / electric-sql/electric

Enhancement: Coordinate retry backoff across ShapeStream instances

Open
#3,897 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
10.4k
Forks
375
Avg merge
3d 1h
Merged PRs (30d)
18

Description

## Problem

When a proxy in front of Electric fails, each `ShapeStream` instance retries independently with its own backoff state. Even with improved backoff defaults (#3896), 20+ shapes each independently ramping up to 60s max delay means unnecessary aggregate load and slow recovery — if one stream discovers the backend is healthy, the others don't find out until their own next retry fires.

## Proposed solution

Add a shared "connectivity signal" that `ShapeStream` instances can opt into. When any stream gets a successful response, it broadcasts a signal that other streams can listen for to skip their current backoff and retry immediately.

### Sketch

```typescript
// Shared signal — one per app, passed to all ShapeStream instances
const connectivitySignal = new EventTarget()

// In ShapeStreamOptions
interface ShapeStreamOptions {
// ...existing options
connectivitySignal?: EventTarget
}

// On successful response (in #requestShapeLongPoll / #requestShapeSSE):
this.options.connectivitySignal?.dispatchEvent(new Event('connected'))

// During backoff wait:
// Listen for 'connected' event to resolve the backoff delay early
```

This is essentially a **circuit breaker** pattern — the circuit is "open" (backing off) when the proxy is down, and when one stream "closes" it by succeeding, all others benefit immediately.

### Considerations

- **Opt-in**: Only streams that share the same `connectivitySignal` coordinate. Streams targeting different backends should use different signals (or none).
- **Thundering herd mitigation**: When the signal fires, streams should add a small random jitter (e.g., 0-500ms) before retrying, rather than all retrying at the exact same instant.
- **Reset backoff state**: On receiving the signal, streams should reset their backoff counter so they don't immediately jump back to a long delay if the next request also fails.
- **Scope**: This only helps when multiple streams share the same backend/proxy. It's a no-op for single-stream apps.

## Related

- #3896 — Increases default backoff parameters (immediate fix)
- #3895 — Tracks the `onError` retry tight-loop bug

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.