Azure / Azure/unbounded

Gantry: 3 - Shutdown pump stop gat

Open
#279 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
28
Forks
11
Avg merge
1d 8h
Merged PRs (30d)
55

Description

## Scope

Fix the puller-pump WaitGroup Add/Wait shutdown race.

## Problem

Shutdown waits on `pullerPumpWG` while new pump starts can still call `wg.Add(1)`. This can panic or wedge during normal rollouts.

Evidence:

- `cmd/gantry/agent_shutdown.go:82`
- `cmd/gantry/main.go:1996`

## Changes

- Add a pump-level stop gate serialized with `wg.Add`.
- The pump should check an accepting flag under a mutex before starting work.
- Shutdown flips the flag to closed before calling `pullerPumpWG.Wait()`.
- After the flag is closed, the pump returns an internal declined outcome and never calls `wg.Add`.
- Unbind coord before waiting as defense in depth.
- Add `coord.Server.Unbind(h)` or a stop hook that calls `h.RemoveStreamHandler(coord.ProtocolID)`.
- Do not rely on unbinding alone: active handlers and local prefetch can still reach the pump until the stop gate is closed.
- Tie `firePrefetch` and `reAdvertiseDigest` to the agent lifecycle in a later lifecycle PR.

## Tests

- Pump refuses new work after shutdown stop gate closes.
- Concurrent `Wait()` and late `please_pull` cannot trigger `wg.Add` after wait starts.
- Coord handler is unbound before pump wait in graceful shutdown.
- Race detector on a shutdown-under-load test.

Contributor guide

Open the contributing guide

Research direction

Start with cmd/gantry/agent_shutdown.go:82 and cmd/gantry/main.go:1996, then trace pullerPumpWG, the puller-pump start path, and the coord handler. Run the existing shutdown-under-load tests with the race detector if available. Done means shutdown closes the stop gate and unbinds coord before waiting, while late please_pull requests are declined without calling wg.Add.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.