api: "Failed to place sandbox" gives no diagnostic - expose why no node accepted the sandbox
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 1.6k
- Forks
- 438
- PR merge metrics
- No merged PRs in 30d
Description
Problem
When sandbox creation fails because no node can host it, the client receives a generic 500 error:
This tells the user nothing about why placement failed. The actual cause is buried in server logs as an internal error and is never surfaced. Common root causes include:
- CPU model mismatch: the template was built on a newer CPU generation (e.g. Emerald Rapids, model 207) but all cluster nodes run an older generation (e.g. Ice Lake, model 106). Cross-generation placement is asymmetric — an n2 build can resume on an n4 node, but not the reverse.
- Label mismatch: the team or template requires scheduling labels (e.g.
gpu,fast-disk) that no available node carries. - All nodes not-accepting: nodes are draining, unhealthy, or saturated.
Without seeing which filter eliminated all nodes it is impossible to self-diagnose or file a useful bug report.
Root cause
BestOfK.chooseNode returns a FailedToPlaceSandboxError whose Error() string includes machine=... and labels=... constraints, but create_instance.go stores that in the internal Err field and returns a hardcoded ClientMsg: "Failed to place sandbox" that never reaches the client.
Proposed fix
- Track per-filter rejection counts inside
BestOfK.sample()(not-accepting, cpu-incompatible, label-filtered, excluded). - Embed those counts plus the build CPU constraints in
FailedToPlaceSandboxError.Error(). - Propagate
err.Error()toClientMsgincreate_instance.go.
Example new error message:
Failed to place sandbox: no compatible node found (38 nodes checked: 0 not-accepting, 38 cpu-incompatible, 0 label-filtered, 0 excluded); build cpu: arch=x86_64 family=6 model=207
This lets the caller immediately identify a CPU-generation mismatch or label gap without digging through server logs.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with BestOfK.sample and BestOfK.chooseNode to trace how node rejection is recorded, then inspect create_instance.go where FailedToPlaceSandboxError is stored and the client message is set. Done means the client receives rejection counts and build CPU constraints that explain why no node accepted the sandbox.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- infrastructure
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100