NVIDIA / NVIDIA/nvcf

Cold-start 503 burst after idle: root cause

Open
#1,535 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
218
Forks
72
Avg merge
1d 9h
Merged PRs (30d)
417

Description

Description

The first ~5-10 s of traffic after an idle period returns a burst of
empty-body 503s, then self-clears and never recurs while traffic stays warm.
Observed at 4 of 5 load starts during staging chaos tests (stargate 0.14.4);
0.1-0.2% of a sustained run. Root cause is not concluded.

Why it matters: first-request-after-idle is the normal case for low-traffic
functions, so this hits exactly the demo/onboarding path.

Investigation state: the 503s carry no body and no router log line. Ruled
out: admission rejection and no-eligible-candidates (both produce JSON
bodies and warnings). Remaining candidates: bare 503 finalization on stale
post-idle estimates, or retry-exhausted pass-through of a bodyless upstream
503/429. Counters that discriminate: admission rejections, 503 responses,
and retry-exhaustion totals during a reproduced burst.

Definition of Done

  • Root cause identified and written up.
  • Either fixed, or explicitly accepted with the behavior documented.

By submitting this issue, you acknowledge that you are an assigned member of the NVCF development team and agree to follow our code of conduct and our contributing guidelines.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the cold-start burst in the staging chaos test and correlate admission rejections, 503 responses, and retry-exhaustion totals. Compare the two remaining candidates: stale post-idle estimates versus retry-exhausted bodyless upstream 503/429 responses. Done means the root cause is written up and the behavior is either fixed or explicitly documented as accepted.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, cloud, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.