Cold-start 503 burst after idle: root cause
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 218
- Forks
- 72
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 417
Description
Description
The first ~5-10 s of traffic after an idle period returns a burst of
empty-body 503s, then self-clears and never recurs while traffic stays warm.
Observed at 4 of 5 load starts during staging chaos tests (stargate 0.14.4);
0.1-0.2% of a sustained run. Root cause is not concluded.
Why it matters: first-request-after-idle is the normal case for low-traffic
functions, so this hits exactly the demo/onboarding path.
Investigation state: the 503s carry no body and no router log line. Ruled
out: admission rejection and no-eligible-candidates (both produce JSON
bodies and warnings). Remaining candidates: bare 503 finalization on stale
post-idle estimates, or retry-exhausted pass-through of a bodyless upstream
503/429. Counters that discriminate: admission rejections, 503 responses,
and retry-exhaustion totals during a reproduced burst.
Definition of Done
- Root cause identified and written up.
- Either fixed, or explicitly accepted with the behavior documented.
By submitting this issue, you acknowledge that you are an assigned member of the NVCF development team and agree to follow our code of conduct and our contributing guidelines.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the cold-start burst in the staging chaos test and correlate admission rejections, 503 responses, and retry-exhaustion totals. Compare the two remaining candidates: stale post-idle estimates versus retry-exhausted bodyless upstream 503/429 responses. Done means the root cause is written up and the behavior is either fixed or explicitly documented as accepted.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, cloud, observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100