Azure / Azure/unbounded

Gantry: 4 - Bound please_pull and pump fan-out

Open
#280 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
28
Forks
11
Avg merge
1d 8h
Merged PRs (30d)
55

Description

## Scope

Bound `please_pull` and pump fan-out. Do not mix this with peer authorization or repository validation.

## Problem

`please_pull` has no digest cap, discards the dispatch context in the pump path, and has no global pull cap. This allows authorized-but-buggy peers, compromised in-mesh peers, or malformed internal callers to amplify a single request into many background pulls.

Evidence:

- `internal/gantry/coord/coord.go:504`
- `cmd/gantry/main.go:1933`
- `cmd/gantry/main.go:1957`
- `cmd/gantry/main.go:1999`

## Changes

- Add explicit config for coord bounds: `coord_max_digests_per_request` and `coord_max_concurrent_pulls`.
- Derive and document defaults. A starting point is `256` digests/request and `16` concurrent pulls, but final values must be justified against manifest child counts, envelope size, node memory/file-descriptor budgets, and DaemonSet resource limits.
- Cap inbound `please_pull` digest count in `servePleasePull`.
- If `len(req.GetDigests())` exceeds the cap, bump `OnStreamError` and reject the request instead of partially processing it.
- Add client-side chunking in `coord.Client.PleasePull` and prefetch/local callers so legitimate internal batches do not exceed the server cap.
- Respect the dispatch deadline in `servePleasePull` and `StartLocalPull`.
- Use the pump context for the local `Has` probe in `newPullerPump`.
- Add an internal pump outcome instead of overloading booleans.
- When the global pull semaphore is full, return `pumpDeclined` and map it to `OUTCOME_UNSPECIFIED` on the current wire protocol. Do not map it to `ALREADY_PULLING`.
- Add a global concurrent-pull semaphore in `newPullerPump`.

## Tests

- Oversized remote `please_pull` is rejected and increments stream-error metrics.
- Client-side `PleasePull` chunking respects the cap.
- Cancelled contexts stop remote and local digest loops.
- Pull semaphore caps concurrent pulls; the N+1 request produces `OUTCOME_UNSPECIFIED`, not `ALREADY_PULLING`.
- Fuzz or property tests cover malformed and oversized `PleasePullRequest` envelopes.

Contributor guide

Open the contributing guide

Research direction

Begin with internal/gantry/coord/coord.go:504 and the servePleasePull, StartLocalPull, and newPullerPump entry points in cmd/gantry/main.go. Trace dispatch contexts, client and prefetch callers, and the current wire outcome mapping before deciding how the two bounds and semaphore interact. Done means the listed rejection, chunking, cancellation, semaphore, metric, and malformed-envelope tests pass without changing authorization or repository validation.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend-api-design, distributed-systems, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.