electric-sql / electric-sql/electric

Ensure a given Electric instance holds the lock to operate on publication manager

Open
#3,286 4 comments 0 reactions 0 assignees View on GitHub
triage
Dominant language
TypeScript
Stars
10.4k
Forks
375
Avg merge
3d 1h
Merged PRs (30d)
18

Description

**Versions**
- Electric: a057f9c7dc07743c8be687ba5e45ce17fb9172db

**Bug description**
At any point our lock connection might get lost, due to network partitions or anything else, and another Electric B waiting (e.g. in a rolling deploy) can immediately grab it and start operating on the slot and publication.

When the lock connection is lost in Electric A, until everything gets shut down and restarted, it might still be operating on the slot and publication without a lock, causing inconsistencies across the two Electrics.

**Expected behavior**
No modifying operations should be made by an Electric if it's not holding the lock.

**Suggested solution**

- Replication connection modifications
- Solved by moving the lock inside the replication connection https://github.com/electric-sql/stratovolt/issues/811
- Replication slot dropping on cleanup
- Since we do this in the connection manager, we can ensure the lock is alive before doing that, otherwise the same solution as for the publication manager below
- Publication manager modifications
- Before any modification to the publication, check that 1) the lock is active, and 2) it is owned by the stack itself, either by matching the backend pid or investigate if the advisory lock can be given some extra metadata to match on.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.