porsager / porsager/postgres

reserve() permanently stranded when a pooled connection is terminated server-side while the reserve is queued (3.4.9)

Open
#1,195 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
JavaScript
Stars
8.7k
Forks
374
Avg merge
11d 16h
Merged PRs (30d)
1

Description

Summary

A reserve() that is still queued (waiting for a pooled connection to free) is permanently stranded if the server terminates the connection it was waiting on. The promise returned by reserve() never settles — no resolve, no reject, no timeout — and the reserved slot is lost for the lifetime of the process.

Reproduced deterministically (3/3) on postgres@3.4.9 (current latest) against PostgreSQL 16.6, Node v22.18.0.

Mechanism

reserve() (src/index.js:203) pushes a { reserve, reject } pseudo-query into queries and waits. onopen() is the only place a reserve is resolved — src/index.js:410, if (query.reserve) return query.reserve(c).

When the connection the reserve is waiting on is closed by the server, onclose() runs:

// src/index.js:426
queries.length && connect(c, queries.shift())

This shifts the pending reserve out of queries and hands it to the dead connection's reconnect as initial. On the reconnect, initial being a reserve is dropped rather than resolved:

// src/connection.js:563 (fetch_types path, the default)
initial.reserve && (initial = null)
// src/connection.js:567 (non-fetch_types path)
initial && !initial.reserve && execute(initial)   // reserve is explicitly NOT executed

So after reconnect the reserve is gone from queries (shifted out at index.js:426) and was never resolved (dropped at connection.js:563/567). onopen() fires on the healthy new connection, sees no matching queued reserve, and the original promise hangs forever.

Reproduction

Single file, no framework, no application code. max: 1 makes the queueing deterministic; any pool size reproduces it whenever a reserve is queued behind a connection that then dies.

import postgres from 'postgres'

const base = { host: '127.0.0.1', port: 5499, user: 'repro', database: 'postgres', max: 1 }

const withTimeout = (p, ms, label) =>
  Promise.race([
    p.then((v) => ({ ok: true, v })),
    new Promise((res) => setTimeout(() => res({ ok: false, label, ms }), ms)),
  ])

const sql = postgres(base)
const admin = postgres({ ...base, max: 1 })

// 1. Take the single pooled connection.
const held = await sql.reserve()
const [{ pid }] = await held`select pg_backend_pid()::int as pid`

// 2. Queue a SECOND reserve() — it waits in the pool's `queries` array.
const pendingReserve = sql.reserve()

// 3. Server terminates the held backend (a failover, an idle reaper, or an admin).
await admin`select pg_terminate_backend(${pid})`

// 4. onclose() shifts the pending reserve out of `queries` and hands it to the
//    dead connection's reconnect, which drops it. It never settles.
const outcome = await withTimeout(pendingReserve, 8000, 'pendingReserve')
console.log(outcome.ok
  ? 'pending reserve() RESOLVED — bug not reproduced'
  : `pending reserve() NEVER SETTLED after ${outcome.ms} ms — STRANDED (bug reproduced)`)

await admin.end({ timeout: 1 }).catch(() => {})
process.exit(outcome.ok ? 1 : 0)

Output:

held reserved connection, backend pid=93939
second reserve() is now queued (pending)
server terminated backend pid=93939
RESULT: pending reserve() NEVER SETTLED after 8000 ms — STRANDED (bug reproduced)
Why it matters

reserve() is the documented way to pin a connection for a transaction or a session-scoped operation. A server-side termination of a pooled connection is routine in production — failover, an idle-connection reaper, idle_in_transaction_session_timeout, an admin pg_terminate_backend. When one lands while a reserve is queued, the caller's await sql.reserve() hangs with no error to catch and no timeout to trip, so a request handler waits forever and the reserved slot leaks. There is no application-side workaround short of racing every reserve() against a manual timeout.

Relationship to #751

This looks like the same root cause as #751, seen from the other side. #751 is the first-ever-connect path with fetch_types: false (connection.js returns before onopen); this is the reconnect-after-close path with fetch_types: true (the default). Both share one invariant: a reserve pseudo-query must remain in queries until onopen resolves it, and must never be consumed as a connection's initial. A fix that restores that invariant should close both.

Fix direction

I have a proven red repro but deliberately am not attaching a patch, because the naive one-line change at index.js:426 (reconnect empty instead of shifting the reserve) has to be coordinated with the reserve-drop at connection.js:563/567 and made safe against repeated close events, and that is your call to make in your own reconnect model. The shape that fixed it in my testing is: leave a head-of-queue reserve in queries on close and reconnect the socket empty, so onopen (index.js:410) resolves it the same way it resolves a fresh reserve — rather than routing it through initial, which drops it. Happy to open a PR along those lines if you'd like, and to fold in a regression test built on the repro above.

Versions: postgres 3.4.9, PostgreSQL 16.6, Node 22.18.0, macOS (aarch64). default_pipeline/prepared settings default; reproduced with fetch_types default (true).

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the reserve queue and reconnect paths in src/index.js:203, 410, and 426, then compare the initial-reserve handling at src/connection.js:563 and 567. Use the supplied PostgreSQL termination reproduction as the failing case. Done means the queued reserve settles after reconnect instead of being dropped, with a regression test based on that reproduction.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, nodejs, postgresql
Domain
database
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.