NVIDIA / NVIDIA/nvcf

Support ordered fallbacks for sticky shadow sampling

Open
#1,910 0 comments 0 reactions 1 assignee View on GitHub

@along-2017 is already working on this.

Since Sep 16, 2026.

needs-triage
Dominant language
Go
Stars
218
Forks
72
Avg merge
1d 12h
Merged PRs (30d)
427

Description

Is this related to a problem? Please describe.

Vanity Gateway's default random shadow sampling makes an independent decision for each request. A multi-turn session can therefore enter and leave the shadow sample between turns, reducing the usefulness of shadow traffic when evaluating routing changes.

perBearerKey provides sticky sampling, but one bearer credential may represent many users or sessions. Current scalar sampling configuration also cannot prefer a request-level key and fall back to request content when that key is unavailable.

Describe the solution you'd like

Add two shadow sampling methods for /v1/chat/completions and /v1/responses:

  • promptCacheKey: hash a nonempty string resolved from the request body's prompt_cache_key or a configured request header.
  • firstMessageHash: hash canonical request content representing the leading instructions and first user message.
    • Chat Completions: leading system and developer messages plus the first user message.
    • Responses: instructions plus the first user input. A string input is treated as user input.

The canonical representation must preserve relevant roles, content, and content-part order while remaining stable across JSON whitespace and object-key ordering. Use the existing shadow bucket algorithm: SHA-256, the first eight digest bytes as a big-endian integer, then modulo 100.

Allow both shadows[].samplingMethod and legacy shadowSamplingMethod to contain either one method or an ordered method list:

promptCacheKeyHeaders:
  - x-multi-turn-session-id
shadows:
  - modelName: private/example/shadow
    percentage: 25
    samplingMethod:
      - promptCacheKey
      - firstMessageHash

Evaluate methods in order:

  1. If a method cannot obtain usable source material, continue to the next method.
  2. The first method with usable material computes the bucket and makes the final admission decision. Do not continue when its bucket falls outside the configured percentage.
  3. Treat random as an implicit terminal fallback for every scalar or list.
  4. If random is written explicitly, require it to be last.
  5. Reject empty lists, empty list entries, duplicate methods, unknown methods, and methods after random.

Add route-level promptCacheKeyHeaders configuration to Chat Completions and Responses. It applies to both legacy and per-target shadow configuration. An omitted field defaults to x-multi-turn-session-id; an explicit [] disables header lookup. Reject null, invalid HTTP header names, case-insensitive duplicate names, and the credential-bearing Authorization and Proxy-Authorization headers.

Resolve promptCacheKey source material in this order:

  1. Use a nonempty string prompt_cache_key from the JSON body.
  2. If the body value is unavailable, check configured headers in list order.
  3. A header is usable only when it has exactly one nonempty value after trimming.
  4. The first usable value makes the final admission decision. Do not try another source after a valid bucket rejects admission.

Hash only the resolved key value. The same value must receive the same bucket whether it came from the body or any configured header and whether the request uses Chat Completions or Responses.

A missing, empty, or non-string prompt_cache_key is unavailable for sampling. Missing or empty first-user content is also unavailable. Sampling must not change primary request validation or forwarded request bytes.

The implicit fallback also applies to existing scalar methods. For example, samplingMethod: perBearerKey uses random sampling when bearer credentials are unavailable instead of skipping the shadow target. Document this behavior change.

Keep the implementation stateless. Do not resolve previous_response_id, conversation state, or WebSocket deltas to recover earlier input. Requests that do not carry the initial material proceed to the next configured method.

Reject promptCacheKey, firstMessageHash, and promptCacheKeyHeaders on endpoint sections other than Chat Completions and Responses. Existing random and perBearerKey support remains unchanged elsewhere.

Acceptance criteria

  • Repeated requests with the same prompt cache key receive the same bucket across body and header sources and across both supported endpoints.
  • Body prompt_cache_key takes precedence over configured headers.
  • Header lookup follows configured order, skips missing, empty, or ambiguous headers, and can be disabled explicitly.
  • Authorization and Proxy-Authorization cannot be configured as prompt cache key headers, regardless of case.
  • Adding later conversation turns does not change firstMessageHash.
  • JSON formatting and object-key ordering do not change firstMessageHash.
  • Chat Completions and Responses have fixed hash vectors covering text and structured content.
  • Fallback proceeds only when source material is unavailable, not after a valid method rejects admission.
  • Scalar and list forms work in both legacy and per-target configuration.
  • A 100 percent target remains admitted without parsing sampling material.
  • Raw cache keys and message content never appear in logs, traces, or metric labels.
  • Gateway configuration tests, admission tests, Helm schema tests, and existing shadow tests pass.
  • Service and chart READMEs document the methods, ordered fallback, header precedence, endpoint limits, stateless limitation, and compatibility change.

Describe alternatives you've considered

Continuing to use random samples requests rather than sessions. perBearerKey can group unrelated sessions when credentials are shared. Adding separate fallback fields would expand the configuration surface and make precedence harder to understand.

Additional context

This extends the per-target shadow policy work from #1243 and #1670.

No new architecture diagram is needed. Update the existing shadow configuration tables, examples, and behavior documentation.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.