jackc / jackc/pgx

Detect stuck TCP connections

Open
#2,163 6 comments 1 reaction 0 assignees View on GitHub
Dominant language
Go
Stars
14.3k
Forks
1.1k
Avg merge
6d 9h
Merged PRs (30d)
11

Description

**Is your feature request related to a problem? Please describe.**

When we failover PostgreSQL server to another primary, old connections can remain stuck: for whatever reason they are not closed and attempt to use them has end up with timeout:

```
write failed: write tcp 10.32.14.13:36102->10.36.54.61:5432: i/o timeout
```

we see these entries minutes after failover with _all_ connections in connection pool

**Describe the solution you'd like**

It seems that detecting hung TCP connection is not [trivial](https://github.com/golang/go/issues/31490). TCP keepalive won't help, because with outstanding writes connection is marked as active and no keepalives are set.

One way I can think of is if TCP IO timeout was somehow decoupled from Query context timeout. [net.Conn](https://pkg.go.dev/net#Conn) has `SetWriteDeadline(t time.Time) error` method which on the surface should do the trick: configurable short timeout of 1-3 seconds will be enough to detect IO error and take connection out of the pool still allowing Query context have arbitrary timeout.

**Describe alternatives you've considered**

Have custom Dialer which sets `TCP_USER_TIMEOUT` (not unlike `tcp_user_timeout` connection string param in [libpq](https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-TCP-USER-TIMEOUT)) with raw sys calls, but it is not portable.

Another options is pgxpool connection health check. It seems to check connection health on `Acquire`, which underneath executes `-- ping` statement, but context passed for that check is from `Acquire` call which in turn itself is from pgxpool.Query so it suffers from the same problem of stuck connection: context timeout is tuned for query execution time overall. If health check was executed with a smaller context timeout derived from parent context then it would detect problematic connections more reliably. This is not a preferred option because it doesn't handle stuck connection detection for already Acquired connections.

Contributor guide

Open the contributing guide

Research direction

Start by reading the pgxpool Acquire and Query health-check paths described in the issue, then inspect how net.Conn deadlines are applied. Compare the health-check context with the query context and investigate the SetWriteDeadline and TCP_USER_TIMEOUT alternatives. Done should include a reproducible failover case where stuck connections are detected without imposing the same short timeout on query execution.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, postgresql
Domain
backend-api-design, database
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.