fix(sandbox): eliminate startup stale-policy forward proxy race
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 8.7k
- Forks
- 1.3k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 253
Description
Description
PR #1929 deflakes the forward proxy GraphQL L7 e2e by retrying expected-allowed requests that temporarily return 403. The linked review explains the root cause: startup symlink-resolution policy reload can run after the entrypoint PID and /proc/<pid>/root become available, advancing the policy generation while the first allowed HTTP forward request is being evaluated. That lets an allowed request observe a stale generation and receive a transient 403 even though the L7 policy should allow it.
Review discussion: https://github.com/NVIDIA/OpenShell/pull/1929#discussion_r3423481489
Context
The PR workaround is intentionally test-scoped: it retries only requests that the test expects to allow, and expected-denied requests stay single-shot. That avoids masking too-permissive-policy regressions, but it does not address the product race.
Root-cause directions captured in review:
- Add explicit startup readiness, for example a narrow internal
policy.localendpoint such asGET http://policy.local/v1/policy/startup-ready?timeout=10, which resolves once the initial symlink-resolution reload has succeeded, failed non-fatally, or been skipped. - Or handle the race inside the forward proxy: for one-shot HTTP forward requests, if policy generation changes before any bytes are written upstream, re-evaluate once against the current generation instead of returning a transient
403. - Account for GraphQL and chunked request bodies carefully. Buffered inspection must not be reread or replayed after an upstream write.
Definition of Done
- Choose whether readiness signaling, proxy-side re-evaluation, or another design is the product fix.
- Implement the fix without hiding expected-denied L7 regressions.
- Keep or update e2e coverage so expected-allowed startup requests no longer need ad hoc retry logic.
- Document any new internal diagnostic API or forward proxy generation-retry semantics if introduced.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading PR #1929 and the linked review discussion, then trace the forward proxy startup policy-reload path and the GraphQL L7 e2e coverage. Compare readiness signaling with proxy-side generation re-evaluation, accounting for chunked and buffered request bodies. Done means the product race is fixed, expected-denied regressions remain visible, and startup allowed requests no longer need ad hoc retries.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- graphql, rust
- Domain
- backend-api-design, networking, testing
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100