orchestrator: ResourceExhausted error messages don't include current vs max counts
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 1.6k
- Forks
- 438
- PR merge metrics
- No merged PRs in 30d
Description
Problem
When the orchestrator returns a ResourceExhausted gRPC error, the API logs it as:
WARN Node exhausted, trying another node {sandboxId: ..., nodeId: ..., error: ...}
However the error field only carries a generic message with no actionable numbers, making it hard to tell from logs why placement was rejected. There are three ResourceExhausted sites:
1. Max running sandboxes (sandboxes.go)
runningSandboxes := s.sandboxFactory.Sandboxes.Count()
if runningSandboxes >= maxRunningSandboxesPerNode {
return nil, status.Errorf(codes.ResourceExhausted,
"max number of running sandboxes on node reached (%d), please retry",
maxRunningSandboxesPerNode) // ← max only, current omitted
}
runningSandboxes is right there in scope but isn't included in the error.
2. Too many sandboxes starting (sandboxes.go — TryAcquire branch)
acquired := s.startingSandboxes.TryAcquire(1)
if !acquired {
return nil, status.Errorf(codes.ResourceExhausted,
"too many sandboxes starting on this node, please retry") // ← no counts at all
}
3. Too many sandboxes resuming (utils.go — Acquire / resume branch)
err := s.startingSandboxes.Acquire(ctx, 1)
if err != nil {
return status.Errorf(codes.ResourceExhausted,
"too many sandboxes resuming on this node, please retry") // ← no counts at all
}
Impact
Without current/max values in the error, the "Node exhausted" log line carries no information about how exhausted the node is, making it impossible to distinguish between:
- A node that is truly at capacity (e.g.
current=50, max=50) - A node that hit the semaphore limit for in-flight starts (e.g.
current=10, max=10) while many slots remain
This matters during placement: the API retries across nodes when it sees ResourceExhausted, and there is no structured signal to tell whether the whole fleet is saturated or just a subset of nodes.
Fix
- Add
Current() int64andLimit() int64methods toAdjustableSemaphoreinpackages/shared/pkg/utils/resizable_semaphore.go - Update all three
ResourceExhaustederror messages to includecurrent=N, max=M
Example after fix:
"max number of running sandboxes on node reached: current=50, max=50, please retry"
"too many sandboxes starting on this node: current=10, max=10, please retry"
"too many sandboxes resuming on this node: current=10, max=10, please retry"
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading packages/shared/pkg/utils/resizable_semaphore.go, then inspect the three ResourceExhausted sites in sandboxes.go and utils.go. Add the semaphore count methods and update each error to report current and max values; done means all three messages include those counts.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 74/100