github / github/artifact-attestations-opa-provider

Investigate concurrent validation to avoid serial fetches sharing one request timeout budget

Open
#195 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
26
Forks
13
Avg merge
18h 43m
Merged PRs (30d)
23

Description

## Summary

When a single admission request contains multiple images (e.g. a multi-container pod), the provider validates them **sequentially**, and within each image it fetches the referenced attestation bundles **sequentially**. All of this work shares **one parent context** — the inbound request context, whose deadline is effectively the admission webhook timeout. As a result, a slow fetch for one image can consume most or all of the shared budget and starve the images processed after it.

This issue proposes investigating **concurrent validation** (across images, and across a single image's attestation bundles) so that the total wall-clock time for a request approaches the **max** of its constituent fetches rather than the **sum**.

## Background: two levels of serialization

Gatekeeper's external-data protocol sends a list of image references (keys) in a single request, and the provider validates each in turn:

1. **Images within a request are validated serially.** `Validate` loops over `r.Request.Keys` and calls `BundleFromName(ctx, …)` for each image, passing the same request `ctx` every time (`pkg/provider/provider.go`).
2. **Attestation bundles within an image are fetched serially.** For each image, `DoBundleFromName` iterates the referrers index and, for every matching bundle, performs two registry round-trips (the bundle's manifest `GET` and its blob `GET`) one at a time (`pkg/fetcher/bundle.go`). An image with _N_ attestations therefore costs roughly `2 + 2N` sequential round-trips.

## How this interacts with `--bundle-timeout`

`--bundle-timeout` (default `3s`) is applied **per attempt, per image**: `retryBundle` wraps each attempt in `context.WithTimeout(ctx, timeout)`, and each image gets its own retry loop with up to `--bundle-max-attempts` (default `3`) attempts. So in isolation an image can take up to `max-attempts × bundle-timeout`.

The important subtlety: that per-image budget is **derived from the shared request `ctx`**. `context.WithTimeout(parent, d)` fires at the *earlier* of `now + d` and the parent's deadline. Because every image in the request shares the same parent context (carrying the admission webhook deadline), the per-image timeouts don't compose additively — they're all clamped by the one request-wide deadline. Concretely:

- The images are processed one after another under a single shared deadline.
- If image `#1`'s fetch is slow (e.g. it has many attestations, or the registry is under load), it can consume most of the shared budget.
- Images `#2`, `#3`, … then start with little or no time left, and their first attempt is cancelled almost immediately — they fail not because *their* attestations were slow, but because an earlier image in the same request exhausted the shared budget.

A consequence worth calling out: **raising `--bundle-timeout` does not give a multi-image request more total time** — it's already bounded by the request/webhook deadline that all images share. It only changes how a single image's budget is subdivided across attempts.

## Why it matters

- Multi-container pods (app + sidecars) are common, so multi-image requests are a meaningful fraction of traffic.
- Images vary widely in how many attestations are attached; an image with many attestations pays the `2 + 2N` serial cost and is disproportionately latency-sensitive.
- Under registry latency spikes, the serial-within-serial structure multiplies exposure: every extra round-trip is another chance to cross the deadline, and the cost is borne not just by the slow image but by its request-mates.

## Ideas to investigate

1. **Validate images within a request concurrently** (bounded parallelism), so total time ≈ max(image fetch times) instead of the sum. This directly protects request-mates from one slow image.
2. **Fetch a single image's attestation bundles concurrently** (bounded parallelism) in the referrers loop, reducing an image's own `2 + 2N` serial latency.
3. **Give each image a fair share of the budget**, or a per-image sub-deadline, so one image cannot silently consume the whole request budget.
4. Consider the interaction with `--bundle-timeout` / `--bundle-max-attempts` semantics and document how they behave for multi-image requests.

## Open questions

- What's a safe bound on concurrency (per request and per image) given registry rate limits and provider CPU/memory?
- Should the per-attempt timeout remain per-image, or should there be an explicit per-image sub-budget carved from the request deadline?
- How does concurrency interact with retries and with the in-memory bundle cache?
- Are there ordering or fairness guarantees we want when the shared deadline is tight (e.g. start all images at once vs. round-robin)?

## Relevant code

- `pkg/provider/provider.go` — `Validate` loops over request keys, sequential per-image validation on the shared request context.
- `pkg/fetcher/bundle.go` — `retryBundle` (per-attempt `context.WithTimeout`), `DoBundleFromName` (sequential referrers/bundle fetch, `2 + 2N` round-trips).
- `cmd/aaop/aaop.go` — `--bundle-timeout`, `--bundle-max-attempts`, `--bundle-delay` flag definitions.

Contributor guide

Open the contributing guide

Research direction

Start by reading Validate in pkg/provider/provider.go, retryBundle and DoBundleFromName in pkg/fetcher/bundle.go, then review the timeout flags in cmd/aaop/aaop.go. Investigate bounded concurrency, request and per-image deadlines, retries, caching, and registry limits; done means the open questions have a decided design and documented timeout semantics.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, security
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.