electric-sql / electric-sql/electric
Ensure a given Electric instance holds the lock to operate on publication manager
- Dominant language
- TypeScript
- Stars
- 10.4k
- Forks
- 375
- Avg merge
- 3d 1h
- Merged PRs (30d)
- 18
Description
**Versions**
- Electric: a057f9c7dc07743c8be687ba5e45ce17fb9172db
**Bug description**
At any point our lock connection might get lost, due to network partitions or anything else, and another Electric B waiting (e.g. in a rolling deploy) can immediately grab it and start operating on the slot and publication.
When the lock connection is lost in Electric A, until everything gets shut down and restarted, it might still be operating on the slot and publication without a lock, causing inconsistencies across the two Electrics.
**Expected behavior**
No modifying operations should be made by an Electric if it's not holding the lock.
**Suggested solution**
- Replication connection modifications
- Solved by moving the lock inside the replication connection https://github.com/electric-sql/stratovolt/issues/811
- Replication slot dropping on cleanup
- Since we do this in the connection manager, we can ensure the lock is alive before doing that, otherwise the same solution as for the publication manager below
- Publication manager modifications
- Before any modification to the publication, check that 1) the lock is active, and 2) it is owned by the stack itself, either by matching the backend pid or investigate if the advisory lock can be given some extra metadata to match on.
Contributor guide
Assessment
This issue has not been assessed yet.