Acquire: a released resource stays unusable for as long as the releasing goroutine keeps running (~1 ms per contended acquire)
- Dominant language
- Go
- Stars
- 419
- Forks
- 37
- PR merge metrics
- No merged PRs in 30d
Description
## What happens
Under contention, a resource that `Release` hands to a waiter is not used by that waiter until the releasing goroutine blocks, is preempted, or another P steals the waiter. If the releaser goes on doing CPU work after `Release` (decoding, publishing, the rest of a request handler), the resource is held by nobody for that long. With every P busy that is the whole of the releaser's remaining run: ~1 ms per contended acquire in our service, ~1.85 ms in the benchmark below at 2 ms of post-release work, and throughput down by a third.
## Why
`releaseAcquiredResource` releases `acquireSem` (`x/sync/semaphore`), which wakes the first waiter by closing its channel. The runtime readies a goroutine unblocked by a channel operation into the waker's own P as `runnext`, and `runnext` is the last thing a stealing P tries (`stealWork`, on its final pass, after `runqgrab`'s 3 µs pause). So the waiter runs when its waker parks, unless a P is idle. The semaphore is strictly FIFO, so a goroutine that is already running and calls `Acquire` queues behind the sleeping waiter rather than taking the free token; that is what keeps the token idle. `sync.Mutex` avoids the same trap by letting running goroutines barge, switching to FIFO handoff only after a waiter has starved for 1 ms.
## Reproduction
https://gist.github.com/mikluko/57dbb8a46e2d38b9b8b56cb4d7675f77 — one `_test.go`, public API only: a pool of 4, ~100 waiters, a 1 ms parked hold, and per arm the CPU work the releaser does after `Release`. `residue-µs` is `4 × ns/op − hold`, the time per acquire the resource is held by nobody. `go test -bench . -cpu 4`, linux/arm64, Go 1.26.7, puddle v2.2.2:
```
BenchmarkPuddle/work=0-4 341320 ns/op 5 residue-µs
BenchmarkPuddle/work=1ms-4 381999 ns/op 335 residue-µs
BenchmarkPuddle/work=1ms/yield-4 256290 ns/op 17 residue-µs
BenchmarkPuddle/work=2ms-4 831440 ns/op 1853 residue-µs
BenchmarkPuddle/work=2ms/yield-4 507890 ns/op 43 residue-µs
```
The `yield` arms add `runtime.Gosched()` right after `Release`. `BenchmarkSemaphore` in the gist runs the same load on the bare `x/sync` semaphore and matches arm for arm, so the pool's mutex and idle stack are not involved. darwin/arm64 shows the same shape.
We first met it in pgxpool (v5.10.0): 4 connections shared by 200 workers, `Acquire` at 98% of a hot span, and a flat ~1.1 ms of token time per acquire that no tracer could see inside, found by subtracting the traced connection hold from `MaxConns/λ`. The benchmark reproduces that subtraction.
## Possible fixes
1. `runtime.Gosched()` after `p.acquireSem.Release(1)` in `releaseAcquiredResource` (after the mutex is unlocked): one line, and in our rig it turns the residue into tens of µs and raises throughput 15–80% depending on post-release work. Its cost is that the releaser goes to the global run queue, and a parked hold nearby lengthened by 100–300 µs in the same runs.
2. A barging fast path in `Acquire`: try the semaphore before queueing even when waiters exist, with a starvation guard in the style of `sync.Mutex`. That removes the idle-token window without a yield, at the price of strict FIFO, which x/sync users rely on (golang/go#56910), so it would have to be puddle's own semaphore.
3. Keep the behaviour and document it. We have the yield in our own code, after every release, and it works; the argument for a fix inside the pool is that every pgxpool user under contention pays this and nothing in the pool's stats points at it (`AcquireDuration` includes it as ordinary queueing).
#24 replaced the previous `sync.Cond` design for a scheduler reason too, and the semaphore fixed that pathology; this one is the FIFO handoff it brought in, seen only when the pool is saturated by releasers that keep running.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with releaseAcquiredResource and its p.acquireSem.Release(1) call, then run the linked benchmark with go test -bench . -cpu 4. Compare the semaphore and pool results, and evaluate the listed Gosched, barging, or documentation options against FIFO behavior and the reported throughput and residue measurements.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, performance
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100