electric-sql / electric-sql/pglite

[BUG]: pglite-socket: concurrent extended-protocol batches interleave across connections; one internal throw permanently deadlocks the query queue

Open
#1,046 4 comments 1 reaction 0 assignees View on GitHub
Dominant language
TypeScript
Stars
16k
Forks
442
Avg merge
20h 19m
Merged PRs (30d)
7

Description

**Describe the bug**

Two related defects in `QueryQueueManager.processQueue()` in `pglite-socket` (0.2.7, still present on `main`). Both bite any client that opens more than one connection; together they make a multi-connection test substrate flake under load. Deterministic, minimal repros for each below.

### Defect A — extended-protocol batches from two connections interleave, clobbering the unnamed prepared statement

The queue serializes per *message*, and applies connection affinity only while `db.isInTransaction()`:

https://github.com/electric-sql/pglite/blob/25d0a55e1f1e4c59f26d9e125150dda88a33fd00/packages/pglite-socket/src/index.ts#L82-L98

Outside a transaction there is no affinity at all. But an extended-protocol batch (`Parse`/`Bind`/`Describe`/`Execute`/`Sync`) is not in a transaction between its messages — so when connections A and B send batches concurrently, their messages are dequeued interleaved into the **single shared PGlite session**. B's `Parse` overwrites the unnamed prepared statement between A's `Parse` and A's `Bind`. A then binds against B's statement: bind/type errors, or — when the shapes happen to be compatible — silently wrong rows. Neither client did anything unusual; this is plain non-pipelined use of two connections.

Real Postgres gives each connection its own session, so per-connection unnamed statements can never cross.

**Repro** (`node repro-interleave.mjs`, deps: `@electric-sql/pglite@0.5.4 @electric-sql/pglite-socket@0.2.7 postgres@3.4.9`) — two clients issue differently-shaped one-row SELECTs concurrently; `prepare: false` makes postgres.js use the unnamed statement:

```js
import { PGlite } from '@electric-sql/pglite'
import { PGLiteSocketServer } from '@electric-sql/pglite-socket'
import postgres from 'postgres'

const db = await PGlite.create()
const server = new PGLiteSocketServer({ db, host: '127.0.0.1', port: 0, maxConnections: 8 })
await server.start()
const port = Number(server.getServerConn().split(':').pop())
const url = `postgres://postgres:postgres@127.0.0.1:${port}/postgres`

const a = postgres(url, { max: 1, prepare: false, onnotice: () => {} })
const b = postgres(url, { max: 1, prepare: false, onnotice: () => {} })

try {
for (let i = 0; i < 500; i++) {
const [ra, rb] = await Promise.all([
a`SELECT ${'client-a'}::text AS who, ${i}::int AS i`,
b`SELECT ${i + 1000}::int * 2 AS doubled, ${'x'}::text || 'y' AS s`,
]).catch((err) => {
throw new Error(`iteration ${i}: ${err.message ?? err}`)
})
if (ra[0].who !== 'client-a' || Number(ra[0].i) !== i)
throw new Error(`iteration ${i}: client A got wrong row: ${JSON.stringify(ra[0])}`)
if (Number(rb[0].doubled) !== (i + 1000) * 2 || rb[0].s !== 'xy')
throw new Error(`iteration ${i}: client B got wrong row: ${JSON.stringify(rb[0])}`)
}
console.log('OK: 500 concurrent iterations, no cross-connection interleave')
process.exit(0)
} catch (err) {
console.error('REPRODUCED:', err.message ?? err)
process.exit(1)
}
```

Observed output (fails immediately, every run):

```
REPRODUCED: iteration 0: invalid input syntax for type integer: "client-a"
```

Client A's `'client-a'` text parameter was bound into client B's statement (whose `$1` is an `int`) — direct proof the batches crossed.

**Fix sketch:** treat the first extended-protocol message (`Parse`/`Bind`/`Describe`/`Execute`/`Flush`/`Close`/`Copy*`) from a handler as opening a batch owned by that handler, and dequeue only that handler's messages until its `Sync` completes (releasing ownership on handler disconnect). We run with exactly this patch downstream and it holds up under heavy concurrent load.

### Defect B — one `execProtocolRawStream` throw leaves `processing` stuck `true`, permanently deadlocking the queue for all connections

https://github.com/electric-sql/pglite/blob/25d0a55e1f1e4c59f26d9e125150dda88a33fd00/packages/pglite-socket/src/index.ts#L106-L132

```ts
} catch (error) {
this.log(`query from handler #${query.handlerId} failed:`, error)
query.reject(error as Error)
return // <-- exits processQueue() without ever reaching…
}
...
}
this.processing = false // <-- …this line
```

`enqueue()` only kicks the loop `if (!this.processing)`, so after a single throw **no message from any connection is ever processed again**. New connections' startup messages sit in the queue forever — in a test suite this surfaces as `CONNECT_TIMEOUT` on connections that have nothing to do with the original failure.

SQL errors don't take this path (they come back as `ErrorResponse` bytes), but internal failures do — e.g. a WASM abort, or any in-flight query racing `db.close()` (`PGlite is closed`). Related but distinct from #985: that one is the `idx === -1` wedge with an idle in-transaction handler; this one needs no transaction at all, just one rejected `execProtocolRawStream` call.

**Repro** (`node repro-queue-stall.mjs`) — injects a single rejection at the exact seam (first `Parse` rejects, everything else untouched), then shows a fresh connection against the now-healthy db hangs forever:

```js
import net from 'node:net'
import { PGlite } from '@electric-sql/pglite'
import { PGLiteSocketServer } from '@electric-sql/pglite-socket'
import postgres from 'postgres'

const db = await PGlite.create()

// Fault injection: the FIRST Parse message ('P') rejects, everything else
// behaves normally — simulating a single transient internal failure.
const original = db.execProtocolRawStream.bind(db)
let injected = false
db.execProtocolRawStream = async (message, options) => {
if (!injected && message[0] === 0x50 /* Parse */) {
injected = true
throw new Error('injected internal failure (stand-in for a WASM abort / closed-db race)')
}
return original(message, options)
}

const server = new PGLiteSocketServer({ db, host: '127.0.0.1', port: 0, maxConnections: 8 })
await server.start()
const port = Number(server.getServerConn().split(':').pop())
const url = `postgres://postgres:postgres@127.0.0.1:${port}/postgres`

// --- connection 1: raw socket, trips the injected throw -------------------
function msg(type, body) {
const b = Buffer.concat([Buffer.alloc(4), body])
b.writeInt32BE(body.length + 4, 0)
return type ? Buffer.concat([Buffer.from(type), b]) : b
}
const cstr = (s) => Buffer.from(s + '\0')

await new Promise((resolve) => {
const sock = net.connect(port, '127.0.0.1', () => {
sock.write(msg(null, Buffer.concat([Buffer.from([0, 3, 0, 0]), cstr('user'), cstr('postgres'), cstr('database'), cstr('postgres'), Buffer.from([0])])))
})
let sent = false
sock.on('data', (d) => {
if (!sent && d.includes(0x5a /* ReadyForQuery */)) {
sent = true
sock.write(Buffer.concat([
msg('P', Buffer.concat([cstr(''), cstr('SELECT 1'), Buffer.from([0, 0])])),
msg('S', Buffer.alloc(0)),
]))
}
})
sock.on('close', () => {
console.log('connection 1: socket closed after the injected throw (expected)')
resolve()
})
sock.on('error', () => {})
})

// --- connection 2: fresh client, healthy db — must work on a sane server --
const sql = postgres(url, { max: 1, prepare: false, connect_timeout: 10, onnotice: () => {} })
const outcome = await Promise.race([
sql`SELECT 42 AS answer`.then(
(r) => ({ code: 0, text: `OK: queue recovered, got ${r[0].answer}` }),
(err) => ({ code: 1, text: `REPRODUCED (as error): healthy connection failed after one earlier throw: ${err.message}` }),
),
new Promise((r) => setTimeout(() => r({ code: 1, text: 'REPRODUCED: queue is deadlocked — a healthy connection hangs forever after one earlier throw' }), 12_000)),
])
console.log(outcome.text)
process.exit(outcome.code)
```

Observed output:

```
connection 1: socket closed after the injected throw (expected)
REPRODUCED (as error): healthy connection failed after one earlier throw: write CONNECT_TIMEOUT 127.0.0.1:50849
```

**Fix sketch:** wrap the loop body's failure path so it continues (or wrap the whole loop in `try { … } finally { this.processing = false }`), rejecting only the failed item instead of abandoning the queue.

**Logs**

Included inline above (both scripts print `REPRODUCED: …` and exit 1 on current versions).

**Details**
- PGlite version: `@electric-sql/pglite` 0.5.4, `@electric-sql/pglite-socket` 0.2.7 (both `latest`; defect sites unchanged on `main` @ 25d0a55)
- using any extensions? which ones?: No
- OS version: macOS 26.5.1 (arm64)
- node, bun, deno or browser version: Node 22.18.0; client is postgres.js (`postgres`) 3.4.9 with `prepare: false`

**Additional context**

Found while running a multi-connection integration-test substrate (postgres.js pool → pglite-socket → PGlite). Under load the two defects compound: an interleave from defect A can make `execProtocolRawStream` throw, and defect B then wedges the whole server, surfacing as unrelated `CONNECT_TIMEOUT`s.

Also related (but a separate, already-reported defect): the spurious extra `ReadyForQuery` on errored extended-protocol messages, #958 — I'll add a deterministic repro there.

We currently work around all of the above by patching `processQueue` at test-setup time (batch affinity from first extended-protocol message until `Sync`; `try/finally` on `processing`), and are happy to upstream a PR along those lines if the approach sounds right to the maintainers.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in packages/pglite-socket/src/index.ts at QueryQueueManager.processQueue(), then run repro-interleave.mjs and repro-queue-stall.mjs to observe both failures. Done means extended-protocol messages remain owned by one handler through Sync, and a rejected execProtocolRawStream call does not leave processing stuck so later connections continue to work.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql, typescript
Domain
databases, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.