conjoin opens unbounded concurrent connections — exceeds sshd MaxStartups on git+ssh remotes, so conjoin never completes
- Dominant language
- Go
- Stars
- 24.4k
- Forks
- 873
- Avg merge
- 1d 8h
- Merged PRs (30d)
- 120
Description
### Summary
`conjoinTables` in `go/store/nbs/conjoiner.go` starts one goroutine per conjoinee with no
concurrency limit. For a persister whose `Open` is a network operation this fans out to an
unbounded number of simultaneous connections. Against a **Git remote over SSH** each `Open`
costs its own SSH connection, so a single conjoin exceeds the server's limit on concurrent
*unauthenticated* connections (OpenSSH `MaxStartups`, default `10:30:100`). The server drops
the excess pre-auth, conjoin fails, and the table-file count never comes down.
The push itself succeeds — only the compaction fails — so the store silently degrades.
### Environment
- Transport: Dolt Git remote, `git+ssh://` to a self-hosted bare repo
- Server: OpenSSH with stock `MaxStartups 10:30:100`, `persourcemaxstartups none` (confirmed via `sshd -T`)
- Observed through an application embedding the Dolt Go module (`github.com/dolthub/dolt/go v0.40.5-0.20260715172757-a6690826d767`)
- The unbounded fan-out is unchanged on `main` at time of writing
### Observed behaviour
Every push logs, at warning level:
```
level=info msg="beginning conjoin of database" pkg=store.noms upstream_len=2259
level=warning msg="conjoin of database failed with error"
error="kex_exchange_identification: read: Connection reset by peer
Connection reset by port
fatal: Could not read from remote repository."
Push complete.
```
`kex_exchange_identification` places the failure in **key exchange** — before authentication
and before any payload — which is exactly what `MaxStartups` random-drop produces. Credentials
are not involved: sequential connections succeed 12/12.
### Measurement
Concurrent TCP connections to the remote's SSH port, sampled locally at ~5ms resolution
(5,310 samples) across one conjoin-triggering push:
| Metric | Value |
|---|---|
| Peak concurrent connections | **18** |
| Samples at/above `MaxStartups` threshold of 10 | **72** of 5,310 |
| Total connections opened during the push | **~247** |
| Same metric for a no-op push (no conjoin) | **2** |
Burst shape, consecutive ~5ms samples: `3 → 6 → 9 → 12 → 15 → 15 → 14 → 13 → 8 → 3`. Roughly
150ms wide, and repeated — the threshold is crossed many times within one push, peaking at
nearly double it.
A graduated concurrency probe against the same server reproduces the server-side behaviour
independently: 5 concurrent connections → 0 failures; 15 → 1 failure; 30 → 10 failures.
### Root cause
`go/store/nbs/conjoiner.go`, `conjoinTables`:
```go
eg, ectx := errgroup.WithContext(ctx)
toConjoin := make(chunkSources, len(conjoinees))
for idx := range conjoinees {
i, spec := idx, conjoinees[idx]
eg.Go(func() (err error) {
toConjoin[i], err = p.Open(ectx, spec.name, spec.chunkCount, stats)
return
})
}
```
One goroutine per conjoinee; `SetLimit` is not called anywhere in the file. The fan-out is
bounded only by the size of the conjoinee set, which is itself a function of how far behind
compaction has fallen.
### Why this surfaces now
Unbounded fan-out is harmless for the HTTP remotesapi, where connection pooling and HTTP/2
multiplexing absorb it. Git remotes as Dolt remotes shipped in February 2026, and over
`git+ssh` each `Open` becomes a distinct SSH connection — so the same loop now presents as a
connection storm. The feature's design write-up does not discuss transfer concurrency.
### Impact — it is self-reinforcing
NBS triggers a conjoin once active table files exceed its limit (256). Each failed conjoin
leaves the count higher, so the next attempt has more conjoinees and a wider fan-out, and is
more likely to fail. The store cannot recover unaided. On our store the count has gone
1,962 → 2,113 → 2,259 over three days — roughly 9× the conjoin threshold — and pulls now take
**10+ minutes**, lengthening steadily.
### Proposed fix
Bound the fan-out with `eg.SetLimit(...)`. A default of 8 sits below the stock `MaxStartups`
of 10 and is ample for local persisters, where opening a table file is dominated by the index
read. PR to follow.
Happy to make the limit configurable instead, or to scope it to remote persisters only, if
either is preferred.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.