porsager / porsager/postgres

Clean socket close with a pending initial query never settles it and spins an unbounded zero-delay reconnect loop

Open
#1,193 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
JavaScript
Stars
8.7k
Forks
374
Avg merge
11d 16h
Merged PRs (30d)
1

Description

Summary

A socket that closes cleanly (FIN, hadError === false, no prior ErrorResponse) while a query is parked as a connection's initial leaves that query pending forever, and turns reconnect() into a zero-delay reconnect loop that connect_timeout and max_lifetime cannot bound. Affects 3.4.9 and current master.

Mechanism (src/connection.js, closed(), lines 436-458 on 3.4.9)
idleTimer.cancel()
lifeTimer.cancel()
connectTimer.cancel()        // cancelled on EVERY close ...

socket.removeAllListeners()
socket = null

if (initial)
  return reconnect()         // returns BEFORE every settle path

!hadError && (query || sent.length) && error(...)
closedTime = performance.now()   // never reached on that branch

Three consequences, each verified against the source:

  1. The parked query never settles. A clean close raises no 'error' event and carries no ErrorResponse, so nothing has rejected initial before closed() runs. The if (initial) return reconnect() branch returns before error(...)/errored(...) and before onclose(...), so the query is in no queue and on no timer that can ever settle it. (An RST rejects via the 'error' event; a server ErrorResponse rejects via errored(); only the clean FIN is unhandled.)
  2. closedTime never advances, so reconnect() at line 361-363 (setTimeout(connect, closedTime ? Math.max(0, closedTime + delay - performance.now()) : 0)) fires with delay 0 on every iteration.
  3. No configured bound can fire. connectTimer.cancel() runs on every close and connect() restarts it (line 343); lifeTimer the same (lines 444 / 371). With a sub-millisecond close/reconnect cycle neither ever elapses.
Reproduction (standalone, no Postgres needed)
import net from 'node:net'
import postgres from 'postgres'

let attempts = 0
const server = net.createServer(s => { attempts++; s.end() }) // accept, clean FIN
await new Promise(r => server.listen(0, '127.0.0.1', r))

const sql = postgres(`postgres://u:p@127.0.0.1:${server.address().port}/db`, {
  max: 3, idle_timeout: 20, connect_timeout: 10, prepare: false,
})

const q = sql`select 1`
setTimeout(() => console.log({ attempts }), 8000) // pending, ~5,000+ attempts
await q // never settles

Measured on 3.4.9 / Node 22: the query is still pending at 8 seconds with 5,080 connection attempts (~635 attempts/second, one CPU-bound reconnect loop per parked connection). Once max connections are looping, every further query lands in queries with no onopen or onclose ever coming, so the whole pool stalls.

Real-world trigger: a serverless function whose upstream pooler (Supavisor) closed connections with a clean FIN during a pooler-side event; the pending query rode all the way to the platform's 300-second function ceiling instead of failing at connect_timeout: 10. We measured the pooler's close verb at the socket level: every close it initiates is FIN + hadError=false (an over-limit refusal is preceded by an ErrorResponse, which settles correctly; the FIN itself never does).

Suggested fix

In closed(), treat a close-with-initial like every other close: advance closedTime, ride the shared backoff, and cap consecutive attempts, settling the parked query at the cap:

if (initial) {
  closedTime = performance.now()
  options.shared.retries++
  delay = (typeof backoff === 'function' ? backoff(options.shared.retries) : backoff) * 1000
  if (++closeRetries < 5)          // closeRetries: new per-connection counter,
    return reconnect()             // reset alongside `retries` on ReadyForQuery
  closeRetries = 0
  errored(Errors.connection('CONNECTION_CLOSED', options, socket))
  return onclose(connection, Errors.connection('CONNECTION_CLOSED', options, socket))
}

A transient close still reconnects and recovers (we pin a peer that FINs two handshakes then serves honestly: the parked query resolves), while a peer that closes every attempt rejects the query with CONNECTION_CLOSED after five paced attempts (~1s with the default backoff) instead of hanging forever. Verified against the repro above: rejected in ~800ms after exactly 5 attempts.

We are carrying this as a pnpm patch in production (all three shipped builds) and are happy to open a PR if the approach looks right.

Related: #1192 (same closed()/ReadyForQuery neighbourhood, different defect: the discarded fetchArrayTypes() promise).

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in src/connection.js by reading closed(), reconnect(), connect(), and the ReadyForQuery handling described in the issue. Run the standalone net-server reproduction to observe the clean-FIN loop. Done means a transient close recovers through backoff, while repeated clean closes settle the parked query and stop unbounded reconnect attempts.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, nodejs, postgresql
Domain
backend, databases, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.