NVIDIA / NVIDIA/OpenShell

fix(sandbox): eliminate startup stale-policy forward proxy race

Open
#1,942 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

area:policy area:sandbox gator:follow-up-needed state:stale tech-debt test:e2e topic:l7 topic:testing
Dominant language
Rust
Stars
8.7k
Forks
1.3k
Avg merge
2d 11h
Merged PRs (30d)
253

Description

Description

PR #1929 deflakes the forward proxy GraphQL L7 e2e by retrying expected-allowed requests that temporarily return 403. The linked review explains the root cause: startup symlink-resolution policy reload can run after the entrypoint PID and /proc/<pid>/root become available, advancing the policy generation while the first allowed HTTP forward request is being evaluated. That lets an allowed request observe a stale generation and receive a transient 403 even though the L7 policy should allow it.

Review discussion: https://github.com/NVIDIA/OpenShell/pull/1929#discussion_r3423481489

Context

The PR workaround is intentionally test-scoped: it retries only requests that the test expects to allow, and expected-denied requests stay single-shot. That avoids masking too-permissive-policy regressions, but it does not address the product race.

Root-cause directions captured in review:

  • Add explicit startup readiness, for example a narrow internal policy.local endpoint such as GET http://policy.local/v1/policy/startup-ready?timeout=10, which resolves once the initial symlink-resolution reload has succeeded, failed non-fatally, or been skipped.
  • Or handle the race inside the forward proxy: for one-shot HTTP forward requests, if policy generation changes before any bytes are written upstream, re-evaluate once against the current generation instead of returning a transient 403.
  • Account for GraphQL and chunked request bodies carefully. Buffered inspection must not be reread or replayed after an upstream write.

Definition of Done

  • Choose whether readiness signaling, proxy-side re-evaluation, or another design is the product fix.
  • Implement the fix without hiding expected-denied L7 regressions.
  • Keep or update e2e coverage so expected-allowed startup requests no longer need ad hoc retry logic.
  • Document any new internal diagnostic API or forward proxy generation-retry semantics if introduced.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading PR #1929 and the linked review discussion, then trace the forward proxy startup policy-reload path and the GraphQL L7 e2e coverage. Compare readiness signaling with proxy-side generation re-evaluation, accounting for chunked and buffered request bodies. Done means the product race is fixed, expected-denied regressions remain visible, and startup allowed requests no longer need ad hoc retries.

Written by the indexing model from the issue text.

Assessment

Tech stack
graphql, rust
Domain
backend-api-design, networking, testing
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.