Gantry: 3 - Shutdown pump stop gat
- Dominant language
- Go
- Stars
- 28
- Forks
- 11
- Avg merge
- 1d 8h
- Merged PRs (30d)
- 55
Description
## Scope
Fix the puller-pump WaitGroup Add/Wait shutdown race.
## Problem
Shutdown waits on `pullerPumpWG` while new pump starts can still call `wg.Add(1)`. This can panic or wedge during normal rollouts.
Evidence:
- `cmd/gantry/agent_shutdown.go:82`
- `cmd/gantry/main.go:1996`
## Changes
- Add a pump-level stop gate serialized with `wg.Add`.
- The pump should check an accepting flag under a mutex before starting work.
- Shutdown flips the flag to closed before calling `pullerPumpWG.Wait()`.
- After the flag is closed, the pump returns an internal declined outcome and never calls `wg.Add`.
- Unbind coord before waiting as defense in depth.
- Add `coord.Server.Unbind(h)` or a stop hook that calls `h.RemoveStreamHandler(coord.ProtocolID)`.
- Do not rely on unbinding alone: active handlers and local prefetch can still reach the pump until the stop gate is closed.
- Tie `firePrefetch` and `reAdvertiseDigest` to the agent lifecycle in a later lifecycle PR.
## Tests
- Pump refuses new work after shutdown stop gate closes.
- Concurrent `Wait()` and late `please_pull` cannot trigger `wg.Add` after wait starts.
- Coord handler is unbound before pump wait in graceful shutdown.
- Race detector on a shutdown-under-load test.
Contributor guide
Research direction
Start with cmd/gantry/agent_shutdown.go:82 and cmd/gantry/main.go:1996, then trace pullerPumpWG, the puller-pump start path, and the coord handler. Run the existing shutdown-under-load tests with the race detector if available. Done means shutdown closes the stop gate and unbinds coord before waiting, while late please_pull requests are declined without calling wg.Add.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100