electric-sql / electric-sql/pglite
[BUG]: pglite-socket: concurrent extended-protocol batches interleave across connections; one internal throw permanently deadlocks the query queue
- Dominant language
- TypeScript
- Stars
- 16k
- Forks
- 442
- Avg merge
- 20h 19m
- Merged PRs (30d)
- 7
Description
**Describe the bug**
Two related defects in `QueryQueueManager.processQueue()` in `pglite-socket` (0.2.7, still present on `main`). Both bite any client that opens more than one connection; together they make a multi-connection test substrate flake under load. Deterministic, minimal repros for each below.
### Defect A — extended-protocol batches from two connections interleave, clobbering the unnamed prepared statement
The queue serializes per *message*, and applies connection affinity only while `db.isInTransaction()`:
https://github.com/electric-sql/pglite/blob/25d0a55e1f1e4c59f26d9e125150dda88a33fd00/packages/pglite-socket/src/index.ts#L82-L98
Outside a transaction there is no affinity at all. But an extended-protocol batch (`Parse`/`Bind`/`Describe`/`Execute`/`Sync`) is not in a transaction between its messages — so when connections A and B send batches concurrently, their messages are dequeued interleaved into the **single shared PGlite session**. B's `Parse` overwrites the unnamed prepared statement between A's `Parse` and A's `Bind`. A then binds against B's statement: bind/type errors, or — when the shapes happen to be compatible — silently wrong rows. Neither client did anything unusual; this is plain non-pipelined use of two connections.
Real Postgres gives each connection its own session, so per-connection unnamed statements can never cross.
**Repro** (`node repro-interleave.mjs`, deps: `@electric-sql/pglite@0.5.4 @electric-sql/pglite-socket@0.2.7 postgres@3.4.9`) — two clients issue differently-shaped one-row SELECTs concurrently; `prepare: false` makes postgres.js use the unnamed statement:
```js
import { PGlite } from '@electric-sql/pglite'
import { PGLiteSocketServer } from '@electric-sql/pglite-socket'
import postgres from 'postgres'
const db = await PGlite.create()
const server = new PGLiteSocketServer({ db, host: '127.0.0.1', port: 0, maxConnections: 8 })
await server.start()
const port = Number(server.getServerConn().split(':').pop())
const url = `postgres://postgres:postgres@127.0.0.1:${port}/postgres`
const a = postgres(url, { max: 1, prepare: false, onnotice: () => {} })
const b = postgres(url, { max: 1, prepare: false, onnotice: () => {} })
try {
for (let i = 0; i < 500; i++) {
const [ra, rb] = await Promise.all([
a`SELECT ${'client-a'}::text AS who, ${i}::int AS i`,
b`SELECT ${i + 1000}::int * 2 AS doubled, ${'x'}::text || 'y' AS s`,
]).catch((err) => {
throw new Error(`iteration ${i}: ${err.message ?? err}`)
})
if (ra[0].who !== 'client-a' || Number(ra[0].i) !== i)
throw new Error(`iteration ${i}: client A got wrong row: ${JSON.stringify(ra[0])}`)
if (Number(rb[0].doubled) !== (i + 1000) * 2 || rb[0].s !== 'xy')
throw new Error(`iteration ${i}: client B got wrong row: ${JSON.stringify(rb[0])}`)
}
console.log('OK: 500 concurrent iterations, no cross-connection interleave')
process.exit(0)
} catch (err) {
console.error('REPRODUCED:', err.message ?? err)
process.exit(1)
}
```
Observed output (fails immediately, every run):
```
REPRODUCED: iteration 0: invalid input syntax for type integer: "client-a"
```
Client A's `'client-a'` text parameter was bound into client B's statement (whose `$1` is an `int`) — direct proof the batches crossed.
**Fix sketch:** treat the first extended-protocol message (`Parse`/`Bind`/`Describe`/`Execute`/`Flush`/`Close`/`Copy*`) from a handler as opening a batch owned by that handler, and dequeue only that handler's messages until its `Sync` completes (releasing ownership on handler disconnect). We run with exactly this patch downstream and it holds up under heavy concurrent load.
### Defect B — one `execProtocolRawStream` throw leaves `processing` stuck `true`, permanently deadlocking the queue for all connections
https://github.com/electric-sql/pglite/blob/25d0a55e1f1e4c59f26d9e125150dda88a33fd00/packages/pglite-socket/src/index.ts#L106-L132
```ts
} catch (error) {
this.log(`query from handler #${query.handlerId} failed:`, error)
query.reject(error as Error)
return // <-- exits processQueue() without ever reaching…
}
...
}
this.processing = false // <-- …this line
```
`enqueue()` only kicks the loop `if (!this.processing)`, so after a single throw **no message from any connection is ever processed again**. New connections' startup messages sit in the queue forever — in a test suite this surfaces as `CONNECT_TIMEOUT` on connections that have nothing to do with the original failure.
SQL errors don't take this path (they come back as `ErrorResponse` bytes), but internal failures do — e.g. a WASM abort, or any in-flight query racing `db.close()` (`PGlite is closed`). Related but distinct from #985: that one is the `idx === -1` wedge with an idle in-transaction handler; this one needs no transaction at all, just one rejected `execProtocolRawStream` call.
**Repro** (`node repro-queue-stall.mjs`) — injects a single rejection at the exact seam (first `Parse` rejects, everything else untouched), then shows a fresh connection against the now-healthy db hangs forever:
```js
import net from 'node:net'
import { PGlite } from '@electric-sql/pglite'
import { PGLiteSocketServer } from '@electric-sql/pglite-socket'
import postgres from 'postgres'
const db = await PGlite.create()
// Fault injection: the FIRST Parse message ('P') rejects, everything else
// behaves normally — simulating a single transient internal failure.
const original = db.execProtocolRawStream.bind(db)
let injected = false
db.execProtocolRawStream = async (message, options) => {
if (!injected && message[0] === 0x50 /* Parse */) {
injected = true
throw new Error('injected internal failure (stand-in for a WASM abort / closed-db race)')
}
return original(message, options)
}
const server = new PGLiteSocketServer({ db, host: '127.0.0.1', port: 0, maxConnections: 8 })
await server.start()
const port = Number(server.getServerConn().split(':').pop())
const url = `postgres://postgres:postgres@127.0.0.1:${port}/postgres`
// --- connection 1: raw socket, trips the injected throw -------------------
function msg(type, body) {
const b = Buffer.concat([Buffer.alloc(4), body])
b.writeInt32BE(body.length + 4, 0)
return type ? Buffer.concat([Buffer.from(type), b]) : b
}
const cstr = (s) => Buffer.from(s + '\0')
await new Promise((resolve) => {
const sock = net.connect(port, '127.0.0.1', () => {
sock.write(msg(null, Buffer.concat([Buffer.from([0, 3, 0, 0]), cstr('user'), cstr('postgres'), cstr('database'), cstr('postgres'), Buffer.from([0])])))
})
let sent = false
sock.on('data', (d) => {
if (!sent && d.includes(0x5a /* ReadyForQuery */)) {
sent = true
sock.write(Buffer.concat([
msg('P', Buffer.concat([cstr(''), cstr('SELECT 1'), Buffer.from([0, 0])])),
msg('S', Buffer.alloc(0)),
]))
}
})
sock.on('close', () => {
console.log('connection 1: socket closed after the injected throw (expected)')
resolve()
})
sock.on('error', () => {})
})
// --- connection 2: fresh client, healthy db — must work on a sane server --
const sql = postgres(url, { max: 1, prepare: false, connect_timeout: 10, onnotice: () => {} })
const outcome = await Promise.race([
sql`SELECT 42 AS answer`.then(
(r) => ({ code: 0, text: `OK: queue recovered, got ${r[0].answer}` }),
(err) => ({ code: 1, text: `REPRODUCED (as error): healthy connection failed after one earlier throw: ${err.message}` }),
),
new Promise((r) => setTimeout(() => r({ code: 1, text: 'REPRODUCED: queue is deadlocked — a healthy connection hangs forever after one earlier throw' }), 12_000)),
])
console.log(outcome.text)
process.exit(outcome.code)
```
Observed output:
```
connection 1: socket closed after the injected throw (expected)
REPRODUCED (as error): healthy connection failed after one earlier throw: write CONNECT_TIMEOUT 127.0.0.1:50849
```
**Fix sketch:** wrap the loop body's failure path so it continues (or wrap the whole loop in `try { … } finally { this.processing = false }`), rejecting only the failed item instead of abandoning the queue.
**Logs**
Included inline above (both scripts print `REPRODUCED: …` and exit 1 on current versions).
**Details**
- PGlite version: `@electric-sql/pglite` 0.5.4, `@electric-sql/pglite-socket` 0.2.7 (both `latest`; defect sites unchanged on `main` @ 25d0a55)
- using any extensions? which ones?: No
- OS version: macOS 26.5.1 (arm64)
- node, bun, deno or browser version: Node 22.18.0; client is postgres.js (`postgres`) 3.4.9 with `prepare: false`
**Additional context**
Found while running a multi-connection integration-test substrate (postgres.js pool → pglite-socket → PGlite). Under load the two defects compound: an interleave from defect A can make `execProtocolRawStream` throw, and defect B then wedges the whole server, surfacing as unrelated `CONNECT_TIMEOUT`s.
Also related (but a separate, already-reported defect): the spurious extra `ReadyForQuery` on errored extended-protocol messages, #958 — I'll add a deterministic repro there.
We currently work around all of the above by patching `processQueue` at test-setup time (batch affinity from first extended-protocol message until `Sync`; `try/finally` on `processing`), and are happy to upstream a PR along those lines if the approach sounds right to the maintainers.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in packages/pglite-socket/src/index.ts at QueryQueueManager.processQueue(), then run repro-interleave.mjs and repro-queue-stall.mjs to observe both failures. Done means extended-protocol messages remain owned by one handler through Sync, and a rejected execProtocolRawStream call does not leave processing stuck so later connections continue to work.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- postgresql, typescript
- Domain
- databases, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 52/100