Detect stuck TCP connections
- Dominant language
- Go
- Stars
- 14.3k
- Forks
- 1.1k
- Avg merge
- 6d 9h
- Merged PRs (30d)
- 11
Description
**Is your feature request related to a problem? Please describe.**
When we failover PostgreSQL server to another primary, old connections can remain stuck: for whatever reason they are not closed and attempt to use them has end up with timeout:
```
write failed: write tcp 10.32.14.13:36102->10.36.54.61:5432: i/o timeout
```
we see these entries minutes after failover with _all_ connections in connection pool
**Describe the solution you'd like**
It seems that detecting hung TCP connection is not [trivial](https://github.com/golang/go/issues/31490). TCP keepalive won't help, because with outstanding writes connection is marked as active and no keepalives are set.
One way I can think of is if TCP IO timeout was somehow decoupled from Query context timeout. [net.Conn](https://pkg.go.dev/net#Conn) has `SetWriteDeadline(t time.Time) error` method which on the surface should do the trick: configurable short timeout of 1-3 seconds will be enough to detect IO error and take connection out of the pool still allowing Query context have arbitrary timeout.
**Describe alternatives you've considered**
Have custom Dialer which sets `TCP_USER_TIMEOUT` (not unlike `tcp_user_timeout` connection string param in [libpq](https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-TCP-USER-TIMEOUT)) with raw sys calls, but it is not portable.
Another options is pgxpool connection health check. It seems to check connection health on `Acquire`, which underneath executes `-- ping` statement, but context passed for that check is from `Acquire` call which in turn itself is from pgxpool.Query so it suffers from the same problem of stuck connection: context timeout is tuned for query execution time overall. If health check was executed with a smaller context timeout derived from parent context then it would detect problematic connections more reliably. This is not a preferred option because it doesn't handle stuck connection detection for already Acquired connections.
Contributor guide
Research direction
Start by reading the pgxpool Acquire and Query health-check paths described in the issue, then inspect how net.Conn deadlines are applied. Compare the health-check context with the query context and investigate the SetWriteDeadline and TCP_USER_TIMEOUT alternatives. Done should include a reproducible failover case where stuck connections are detected without imposing the same short timeout on query execution.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, postgresql
- Domain
- backend-api-design, database
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100