electric-sql / electric-sql/electric

Non-blocking writer barrier for publication table additions

Open
#4,788 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
10.4k
Forks
375
Avg merge
3d 1h
Merged PRs (30d)
18

Description

# Non-blocking writer barrier for publication table additions

Follow-up to #4773 / the fix in `Electric.Postgres.Configuration.add_table_to_publication/4` (`wait_for_in_flight_writers/2`).

## Background

Logical decoding evaluates publication membership per change, against the catalog state at that point in the WAL. A transaction that wrote to a table *before* `ALTER PUBLICATION … ADD TABLE` committed never has those writes emitted by the replication stream, even if it commits after. They can only reach a shape through the initial snapshot, which only includes them if the writer has committed by the time the snapshot is taken.

The fix for #4773 closes that gap by taking `LOCK TABLE … IN SHARE MODE` inside the same transaction as the `ADD TABLE`. SHARE conflicts with the ROW EXCLUSIVE lock every writer holds until its transaction ends, so the addition can't commit — and waiting shapes can't start snapshotting — until every in-flight writer has finished. This mirrors what the `ALTER TABLE … REPLICA IDENTITY FULL` on the first-time-add path has always done implicitly through its ACCESS EXCLUSIVE lock. The wait is bounded by the configuration action timeout (5 s by default); on expiry the addition is rolled back and shape creation fails with a retryable error.

## The side effect

SHARE is the weakest lock mode that conflicts with writers, but it also conflicts with SHARE UPDATE EXCLUSIVE, i.e. with `VACUUM`, `ANALYZE`, `CREATE INDEX CONCURRENTLY`, `REINDEX CONCURRENTLY`. While Electric's SHARE request is queued behind one of those, **every new writer on the table queues behind Electric** (Postgres makes new lock requests wait behind conflicting waiters to avoid starvation). Autovacuum cancels itself when it blocks a lock request (except anti-wraparound), but a manual `VACUUM` or a long `CREATE INDEX CONCURRENTLY` won't, so the outcome is:

- shape creation on that table fails after the action timeout with `table_lock_timeout`;
- each attempt stalls all writers on the table for up to that timeout;
- clients retry, so a long-running maintenance operation turns into periodic write stalls for its duration.

The same holds for a genuinely long-running writer (a slow bulk load): shape creation is correctly blocked until it commits, but other writers on the table are stalled for the timeout on every attempt.

None of this is new to Electric — the first-time-add path has had it, with a stronger lock, since the replica identity change was introduced — but the #4773 fix makes it reachable on every re-add (after TRUNCATE invalidation, shape expiry, schema-change invalidation, etc.), not only the first time a table is synced.

## Proposal

Replace the lock with a wait that blocks nobody. Immediately after `ADD TABLE` commits, capture the fixed set of transactions that could have written to the table before the commit, then poll until they have all ended before reporting the relation as configured:

- exact: `pg_locks` rows with `locktype = 'relation'`, `relation = ` (plus partition oids for partitioned tables), mode `RowExclusiveLock` or stronger, keyed by `virtualtransaction`; or
- simpler and slightly over-approximate: the `xip_list` of `pg_current_snapshot()` — only transactions that have written *something* have an xid, so idle read-only sessions are excluded, but writers on unrelated tables are not.

The set is fixed at capture time, so the wait can't starve under continuous write load, doesn't queue anyone, and needs no privilege beyond reading `pg_locks`.

## What it needs

- The natural place is the Configurator, between `do_publication_update(:add, …)` returning and `notify_filters_result(…, {:ok, :configured})`. The Configurator processes relation actions sequentially in one process, so a polling wait there would stall publication updates for every other table; it needs to become a per-relation task with its own deadline.
- That deadline has to be coordinated with the `add_shape` call timeout in the Snapshotter (currently the 5 s `GenServer.call` default → `SnapshotError.table_lock_timeout`).
- Partition enumeration for partitioned tables if the `pg_locks` variant is chosen.
- The `ADD TABLE` transaction would then commit *before* the barrier, so the barrier failing (deadline) must leave the table published and the waiters failed-but-retryable, rather than roll the addition back as the current fix does.

## When to do it

Not urgent. Pick it up if `table_lock_timeout` snapshot errors show up in telemetry at any meaningful rate, or if a user reports write stalls on a table during maintenance operations that correlate with shape creation.

## Related

Separately from Electric, the same "write while unpublished, commit after `ADD TABLE`" transaction triggers a walsender relation-cache bug in PostgreSQL that drops later changes to the table until the next invalidation. It's fixed upstream (PG 18 commit `4909b38af0`, back-patched to supported branches in the August 2025 minor releases — verify against the release notes). The Electric-side barrier doesn't cover it; the docs should recommend running a PG minor that includes the fix.

Contributor guide

Open the contributing guide

Research direction

Start in the Configurator around do_publication_update(:add, …) and notify_filters_result(…, {:ok, :configured}), then trace Snapshotter's add_shape call and its 5 s timeout. Decide how a per-relation barrier and deadline coordinate across these paths, including partition handling if pg_locks is used. Done means writers are not queued, the fixed transaction set is awaited, and timeout failures remain retryable after ADD TABLE commits.

Written by the indexing model from the issue text.

Assessment

Tech stack
elixir, postgresql
Domain
databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.