electric-sql / electric-sql/electric

FlushTracker stall: consumer alive but permanently stuck (no progress detection)

Open
#3,983 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
10.4k
Forks
375
Avg merge
3d 1h
Merged PRs (30d)
18

Description

Parent: #3980

## Scenario

A consumer process is alive (`Process.alive?` returns true) but has stopped making progress on flushing transactions. The FlushTracker tracks the shape, but no `handle_flush_notification` ever arrives because the consumer is stuck.

## How this can happen

- **Deadlock or infinite wait**: The consumer is blocked waiting on a resource that will never become available (e.g., a GenServer.call to a dead process without a timeout, a storage backend that hangs on I/O).
- **Infinite loop in event processing**: A bug in change handling, move-in processing, or materializer interaction causes the consumer to loop without returning from `handle_call`.
- **Message queue starvation**: The consumer's mailbox is flooded with low-priority messages that are processed before the storage `:flushed` callback, effectively starving the flush path indefinitely.

## Why this is distinct

This scenario cannot be detected by process monitoring (Option A in the parent issue) because the consumer process is alive. `Process.alive?` returns true, and no `:DOWN` message is ever sent.

Only a **progress-based detection mechanism** can catch this — e.g., tracking the last time each shape in `FlushTracker.last_flushed` advanced and treating shapes that haven't progressed within a timeout as stuck.

## Fix

Extend the liveness sweep approach (Option B in #3980) with a staleness timeout: if a shape has been in `FlushTracker.last_flushed` for longer than N seconds without its `last_flushed` offset advancing, treat it as stuck and call `handle_shape_removed` (or trigger a consumer restart).

This is the only scenario that requires timeout-based detection rather than monitor-based detection.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.