dolthub / dolthub/dolt

conjoin opens unbounded concurrent connections — exceeds sshd MaxStartups on git+ssh remotes, so conjoin never completes

Open
#11,739 4 comments 0 reactions 1 assignee Claimed by @reltuk View on GitHub
bug customer issue performance
Dominant language
Go
Stars
24.4k
Forks
873
Avg merge
1d 8h
Merged PRs (30d)
120

Description

### Summary

`conjoinTables` in `go/store/nbs/conjoiner.go` starts one goroutine per conjoinee with no
concurrency limit. For a persister whose `Open` is a network operation this fans out to an
unbounded number of simultaneous connections. Against a **Git remote over SSH** each `Open`
costs its own SSH connection, so a single conjoin exceeds the server's limit on concurrent
*unauthenticated* connections (OpenSSH `MaxStartups`, default `10:30:100`). The server drops
the excess pre-auth, conjoin fails, and the table-file count never comes down.

The push itself succeeds — only the compaction fails — so the store silently degrades.

### Environment

- Transport: Dolt Git remote, `git+ssh://` to a self-hosted bare repo
- Server: OpenSSH with stock `MaxStartups 10:30:100`, `persourcemaxstartups none` (confirmed via `sshd -T`)
- Observed through an application embedding the Dolt Go module (`github.com/dolthub/dolt/go v0.40.5-0.20260715172757-a6690826d767`)
- The unbounded fan-out is unchanged on `main` at time of writing

### Observed behaviour

Every push logs, at warning level:

```
level=info msg="beginning conjoin of database" pkg=store.noms upstream_len=2259
level=warning msg="conjoin of database failed with error"
error="kex_exchange_identification: read: Connection reset by peer
Connection reset by port
fatal: Could not read from remote repository."
Push complete.
```

`kex_exchange_identification` places the failure in **key exchange** — before authentication
and before any payload — which is exactly what `MaxStartups` random-drop produces. Credentials
are not involved: sequential connections succeed 12/12.

### Measurement

Concurrent TCP connections to the remote's SSH port, sampled locally at ~5ms resolution
(5,310 samples) across one conjoin-triggering push:

| Metric | Value |
|---|---|
| Peak concurrent connections | **18** |
| Samples at/above `MaxStartups` threshold of 10 | **72** of 5,310 |
| Total connections opened during the push | **~247** |
| Same metric for a no-op push (no conjoin) | **2** |

Burst shape, consecutive ~5ms samples: `3 → 6 → 9 → 12 → 15 → 15 → 14 → 13 → 8 → 3`. Roughly
150ms wide, and repeated — the threshold is crossed many times within one push, peaking at
nearly double it.

A graduated concurrency probe against the same server reproduces the server-side behaviour
independently: 5 concurrent connections → 0 failures; 15 → 1 failure; 30 → 10 failures.

### Root cause

`go/store/nbs/conjoiner.go`, `conjoinTables`:

```go
eg, ectx := errgroup.WithContext(ctx)
toConjoin := make(chunkSources, len(conjoinees))

for idx := range conjoinees {
i, spec := idx, conjoinees[idx]
eg.Go(func() (err error) {
toConjoin[i], err = p.Open(ectx, spec.name, spec.chunkCount, stats)
return
})
}
```

One goroutine per conjoinee; `SetLimit` is not called anywhere in the file. The fan-out is
bounded only by the size of the conjoinee set, which is itself a function of how far behind
compaction has fallen.

### Why this surfaces now

Unbounded fan-out is harmless for the HTTP remotesapi, where connection pooling and HTTP/2
multiplexing absorb it. Git remotes as Dolt remotes shipped in February 2026, and over
`git+ssh` each `Open` becomes a distinct SSH connection — so the same loop now presents as a
connection storm. The feature's design write-up does not discuss transfer concurrency.

### Impact — it is self-reinforcing

NBS triggers a conjoin once active table files exceed its limit (256). Each failed conjoin
leaves the count higher, so the next attempt has more conjoinees and a wider fan-out, and is
more likely to fail. The store cannot recover unaided. On our store the count has gone
1,962 → 2,113 → 2,259 over three days — roughly 9× the conjoin threshold — and pulls now take
**10+ minutes**, lengthening steadily.

### Proposed fix

Bound the fan-out with `eg.SetLimit(...)`. A default of 8 sits below the stock `MaxStartups`
of 10 and is ample for local persisters, where opening a table file is dominated by the index
read. PR to follow.

Happy to make the limit configurable instead, or to scope it to remote persisters only, if
either is preferred.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.