electric-sql / electric-sql/electric
Publication misconfiguration causes infinite restart loop without recovery
- Dominant language
- TypeScript
- Stars
- 10.4k
- Forks
- 375
- Avg merge
- 3d 1h
- Merged PRs (30d)
- 18
Description
## Observed Behavior
While looking through Electric Cloud logs, I noticed a source repeatedly flipping between "waiting" and "active" states. The logs show the following warning cycling continuously:
```
[warning] Publication fatal misconfiguration: %Electric.DbConfigurationError{
type: :publication_missing_operations,
message: "Publication \"...\" does not publish all required operations: INSERT, UPDATE, DELETE, TRUNCATE"
}
Attempting recovery through a restart and fresh setup of connection subsystem.
```
## Investigation
The "restart after failed publication check" behaviour was introduced in #3324. When the publication doesn't satisfy all requirements (INSERT, UPDATE, DELETE, TRUNCATE), the system triggers a connection manager restart:
https://github.com/electric-sql/electric/blob/586bd4c5253a1cbc9b31ddf48510922dc2fe076d/packages/sync-service/lib/electric/replication/publication_manager/configurator.ex#L320-L328
However, the restart alone **does not resolve the issue**. During the connection setup phase, the connection manager checks for an existing publication and, if one exists, skips creation and continues with initialization:
https://github.com/electric-sql/electric/blob/586bd4c5253a1cbc9b31ddf48510922dc2fe076d/packages/sync-service/lib/electric/postgres/replication_client/connection_setup.ex#L166-L172
This creates a loop:
1. Connection manager starts → detects existing publication → continues initialization
2. Publication check runs (~1 minute later) → detects misconfiguration → triggers restart
3. Repeat indefinitely
## Steps to Reproduce
1. Start Electric and wait for it to finish initialization
2. Manually modify the publication Electric created to remove one or more required operations (e.g., remove TRUNCATE)
3. Check logs for the repeating warning message and restarts of the connection subsystem
## Proposed Fix
1. **Run publication validation at initialization** — The connection manager should verify the publication satisfies all requirements during connection setup, not just in the periodic check
2. **Invalidate all shapes on detection** — When this misconfiguration is detected, all existing shapes should be invalidated since we may have missed changes affecting them while the publication was misconfigured
Contributor guide
Assessment
This issue has not been assessed yet.