dolthub / dolthub/dolt

Connection slots leak until Threads_connected pins at max_connections, then the listener dies with the process alive

Open
#11,526 1 comment 0 reactions 0 assignees View on GitHub
customer issue
Dominant language
Go
Stars
24.4k
Forks
873
Avg merge
1d 5h
Merged PRs (30d)
108

Description

### Summary

On dolt 2.2.3, a long-running `dolt sql-server` under a churn-heavy client
workload accumulated connection slots that no socket corresponded to.
`Threads_connected` pinned at exactly `max_connections` while the OS socket
table held roughly 40% of that number. `read_timeout_millis` did not reclaim
the excess. The terminal state was a live server process with no listener bound
to its port at all.

We captured the live state before restarting, because the restart erases it.
Everything below is measured, not inferred.

### The measurement

```
Threads_connected = 256 (read from the server; 256 was max_connections)
real sockets ~104 (22 ESTAB + 41 CLOSE-WAIT + 41 FIN-WAIT-2)
```

About 150 connection slots were held with no socket behind them.
`read_timeout_millis` was 30000. Those slots did not come back.

Once the slot pool is pinned at its ceiling, every new connection goes to the
`back_log` wait queue, which also fills, and the server logs:

```
level=warning msg="max waiting connections reached. Client rejected. Increase server max_connections and back_log"
```

That message names `back_log`, which is the waiting room rather than the
exhausted resource. Following it leads to raising a queue depth in front of a
pool that has no free slots, which converts fast rejections into slow ones.

### Why socket-level observation is misleading here

The socket table read flat at 104 over 45 seconds with CLOSE-WAIT falling. That
looks like congestion reaching equilibrium, and it argues against intervening.
The sockets were flat; the slots were leaking. The two observables disagreed,
and only `Threads_connected` reflected the resource the error was actually
about.

### Terminal state

```
ss -tlnp | grep 29620 -> nothing listening
server pid -> alive, state S, RSS ~1.9GB
sockets on 29620 -> 0, of any state (down from 104)
dolt.log -> last line 20:27:35 -04:00, silent afterwards
other dolt servers -> two other sql-servers on this host still listening
```

Two unrelated `dolt sql-server` processes on the same host kept their listeners,
so this was not host networking.

### Load attribution

We sampled every live client process and walked each to its parent, 251 samples
over 15 seconds:

```
112 diagnostic command, two concurrent runs
117 eight orchestration dispatchers combined
22 a read-only query with --limit=5000
```

The dispatchers were not a runaway. Their own traces show 3122 idle sweeps at a
5s backoff ceiling, so about 1.6 sweeps/sec against roughly 38 connections/sec
measured. Both diagnostic runs then exited on their own, removing about 45% of
arrivals. Availability did not recover. It went to zero and the listener died.
Removing load did not release the slots.

### Ruled out

Not CPU (threads at ~0% instantaneous, host load 2.20 across 16 cores and
falling from 6.54). Not memory. Not file descriptors (1193 of 1048576). Not port
drift. Not a second server on the same port.

### Client workload shape

This is a multi-agent orchestrator ([Gas City](https://github.com/gastownhall/gascity)).
Its CLI opens a short-lived connection per operation and the process frequently
exits without a clean `COM_QUIT`. Connection arrival rate was around 38/sec.
That churn pattern is presumably what surfaces this, and it may be what is
needed to reproduce it.

The history suggests slow accumulation rather than a sudden event: 629,587
`max waiting connections reached` lines had accumulated since 2026-07-17, so the
server degraded for roughly three weeks before it stopped serving.

### Server config

```yaml
listener:
port: 29620
host: 127.0.0.1
max_connections: 256 # since raised to 512, which only buys time
back_log: 50
max_connections_timeout_millis: 5000
read_timeout_millis: 30000
write_timeout_millis: 300000
behavior:
auto_gc_behavior:
enable: true
archive_level: 0
```

`dolt version 2.2.3`, Linux, ~20 databases under one `data_dir`.

### What would help

1. Slots that outlive their sockets get reclaimed, by `read_timeout` or
otherwise. This is the substantive ask.
2. The rejection message names `max_connections` and the observed
`Threads_connected` rather than pointing at `back_log`, since `back_log` is
not the exhausted resource in this state.
3. A server that has lost its listener while the process is still alive is
detectable. Right now an external supervisor cannot distinguish it from a
healthy server without probing the port, and the process itself looks fine.

We do not have a minimal reproduction. What we have is the live state of a
production server at the point of failure, captured deliberately before the
restart. Happy to run further diagnostics against the same workload if there is
something specific worth measuring.

This looks adjacent to #8932 but distinct: that one is about the variables not
being respected, and this is about slots not being released when they are.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at the dolt sql-server connection lifecycle and the max_connections/back_log handling described in the report; reproduce the issue with the cited short-lived-client churn if possible. Compare server connection counts with socket state and observe behavior after the listener disappears. Done means the slot leak and resulting listener failure are reproduced and covered by focused regression checks, with the reported diagnostics addressed.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, sql
Domain
databases, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.