github / github/artifact-attestations-opa-provider
Investigate concurrent validation to avoid serial fetches sharing one request timeout budget
- Dominant language
- Go
- Stars
- 26
- Forks
- 13
- Avg merge
- 18h 43m
- Merged PRs (30d)
- 23
Description
## Summary
When a single admission request contains multiple images (e.g. a multi-container pod), the provider validates them **sequentially**, and within each image it fetches the referenced attestation bundles **sequentially**. All of this work shares **one parent context** — the inbound request context, whose deadline is effectively the admission webhook timeout. As a result, a slow fetch for one image can consume most or all of the shared budget and starve the images processed after it.
This issue proposes investigating **concurrent validation** (across images, and across a single image's attestation bundles) so that the total wall-clock time for a request approaches the **max** of its constituent fetches rather than the **sum**.
## Background: two levels of serialization
Gatekeeper's external-data protocol sends a list of image references (keys) in a single request, and the provider validates each in turn:
1. **Images within a request are validated serially.** `Validate` loops over `r.Request.Keys` and calls `BundleFromName(ctx, …)` for each image, passing the same request `ctx` every time (`pkg/provider/provider.go`).
2. **Attestation bundles within an image are fetched serially.** For each image, `DoBundleFromName` iterates the referrers index and, for every matching bundle, performs two registry round-trips (the bundle's manifest `GET` and its blob `GET`) one at a time (`pkg/fetcher/bundle.go`). An image with _N_ attestations therefore costs roughly `2 + 2N` sequential round-trips.
## How this interacts with `--bundle-timeout`
`--bundle-timeout` (default `3s`) is applied **per attempt, per image**: `retryBundle` wraps each attempt in `context.WithTimeout(ctx, timeout)`, and each image gets its own retry loop with up to `--bundle-max-attempts` (default `3`) attempts. So in isolation an image can take up to `max-attempts × bundle-timeout`.
The important subtlety: that per-image budget is **derived from the shared request `ctx`**. `context.WithTimeout(parent, d)` fires at the *earlier* of `now + d` and the parent's deadline. Because every image in the request shares the same parent context (carrying the admission webhook deadline), the per-image timeouts don't compose additively — they're all clamped by the one request-wide deadline. Concretely:
- The images are processed one after another under a single shared deadline.
- If image `#1`'s fetch is slow (e.g. it has many attestations, or the registry is under load), it can consume most of the shared budget.
- Images `#2`, `#3`, … then start with little or no time left, and their first attempt is cancelled almost immediately — they fail not because *their* attestations were slow, but because an earlier image in the same request exhausted the shared budget.
A consequence worth calling out: **raising `--bundle-timeout` does not give a multi-image request more total time** — it's already bounded by the request/webhook deadline that all images share. It only changes how a single image's budget is subdivided across attempts.
## Why it matters
- Multi-container pods (app + sidecars) are common, so multi-image requests are a meaningful fraction of traffic.
- Images vary widely in how many attestations are attached; an image with many attestations pays the `2 + 2N` serial cost and is disproportionately latency-sensitive.
- Under registry latency spikes, the serial-within-serial structure multiplies exposure: every extra round-trip is another chance to cross the deadline, and the cost is borne not just by the slow image but by its request-mates.
## Ideas to investigate
1. **Validate images within a request concurrently** (bounded parallelism), so total time ≈ max(image fetch times) instead of the sum. This directly protects request-mates from one slow image.
2. **Fetch a single image's attestation bundles concurrently** (bounded parallelism) in the referrers loop, reducing an image's own `2 + 2N` serial latency.
3. **Give each image a fair share of the budget**, or a per-image sub-deadline, so one image cannot silently consume the whole request budget.
4. Consider the interaction with `--bundle-timeout` / `--bundle-max-attempts` semantics and document how they behave for multi-image requests.
## Open questions
- What's a safe bound on concurrency (per request and per image) given registry rate limits and provider CPU/memory?
- Should the per-attempt timeout remain per-image, or should there be an explicit per-image sub-budget carved from the request deadline?
- How does concurrency interact with retries and with the in-memory bundle cache?
- Are there ordering or fairness guarantees we want when the shared deadline is tight (e.g. start all images at once vs. round-robin)?
## Relevant code
- `pkg/provider/provider.go` — `Validate` loops over request keys, sequential per-image validation on the shared request context.
- `pkg/fetcher/bundle.go` — `retryBundle` (per-attempt `context.WithTimeout`), `DoBundleFromName` (sequential referrers/bundle fetch, `2 + 2N` round-trips).
- `cmd/aaop/aaop.go` — `--bundle-timeout`, `--bundle-max-attempts`, `--bundle-delay` flag definitions.
Contributor guide
Research direction
Start by reading Validate in pkg/provider/provider.go, retryBundle and DoBundleFromName in pkg/fetcher/bundle.go, then review the timeout flags in cmd/aaop/aaop.go. Investigate bounded concurrency, request and per-image deadlines, retries, caching, and registry limits; done means the open questions have a decided design and documented timeout semantics.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, security
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100