electric-sql / electric-sql/electric

Publication misconfiguration causes infinite restart loop without recovery

Open
#3,810 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
10.4k
Forks
375
Avg merge
3d 1h
Merged PRs (30d)
18

Description

## Observed Behavior

While looking through Electric Cloud logs, I noticed a source repeatedly flipping between "waiting" and "active" states. The logs show the following warning cycling continuously:

```
[warning] Publication fatal misconfiguration: %Electric.DbConfigurationError{
type: :publication_missing_operations,
message: "Publication \"...\" does not publish all required operations: INSERT, UPDATE, DELETE, TRUNCATE"
}

Attempting recovery through a restart and fresh setup of connection subsystem.
```

## Investigation

The "restart after failed publication check" behaviour was introduced in #3324. When the publication doesn't satisfy all requirements (INSERT, UPDATE, DELETE, TRUNCATE), the system triggers a connection manager restart:

https://github.com/electric-sql/electric/blob/586bd4c5253a1cbc9b31ddf48510922dc2fe076d/packages/sync-service/lib/electric/replication/publication_manager/configurator.ex#L320-L328

However, the restart alone **does not resolve the issue**. During the connection setup phase, the connection manager checks for an existing publication and, if one exists, skips creation and continues with initialization:

https://github.com/electric-sql/electric/blob/586bd4c5253a1cbc9b31ddf48510922dc2fe076d/packages/sync-service/lib/electric/postgres/replication_client/connection_setup.ex#L166-L172

This creates a loop:
1. Connection manager starts → detects existing publication → continues initialization
2. Publication check runs (~1 minute later) → detects misconfiguration → triggers restart
3. Repeat indefinitely

## Steps to Reproduce

1. Start Electric and wait for it to finish initialization
2. Manually modify the publication Electric created to remove one or more required operations (e.g., remove TRUNCATE)
3. Check logs for the repeating warning message and restarts of the connection subsystem

## Proposed Fix

1. **Run publication validation at initialization** — The connection manager should verify the publication satisfies all requirements during connection setup, not just in the periodic check

2. **Invalidate all shapes on detection** — When this misconfiguration is detected, all existing shapes should be invalidated since we may have missed changes affecting them while the publication was misconfigured

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.