e2b-dev / e2b-dev/runtime

orchestrator: ResourceExhausted error messages don't include current vs max counts

Open
#3,583 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
1.6k
Forks
438
PR merge metrics
No merged PRs in 30d

Description

Problem

When the orchestrator returns a ResourceExhausted gRPC error, the API logs it as:

WARN  Node exhausted, trying another node  {sandboxId: ..., nodeId: ..., error: ...}

However the error field only carries a generic message with no actionable numbers, making it hard to tell from logs why placement was rejected. There are three ResourceExhausted sites:

1. Max running sandboxes (sandboxes.go)
runningSandboxes := s.sandboxFactory.Sandboxes.Count()
if runningSandboxes >= maxRunningSandboxesPerNode {
    return nil, status.Errorf(codes.ResourceExhausted,
        "max number of running sandboxes on node reached (%d), please retry",
        maxRunningSandboxesPerNode)   // ← max only, current omitted
}

runningSandboxes is right there in scope but isn't included in the error.

2. Too many sandboxes starting (sandboxes.goTryAcquire branch)
acquired := s.startingSandboxes.TryAcquire(1)
if !acquired {
    return nil, status.Errorf(codes.ResourceExhausted,
        "too many sandboxes starting on this node, please retry")  // ← no counts at all
}
3. Too many sandboxes resuming (utils.goAcquire / resume branch)
err := s.startingSandboxes.Acquire(ctx, 1)
if err != nil {
    return status.Errorf(codes.ResourceExhausted,
        "too many sandboxes resuming on this node, please retry")  // ← no counts at all
}

Impact

Without current/max values in the error, the "Node exhausted" log line carries no information about how exhausted the node is, making it impossible to distinguish between:

  • A node that is truly at capacity (e.g. current=50, max=50)
  • A node that hit the semaphore limit for in-flight starts (e.g. current=10, max=10) while many slots remain

This matters during placement: the API retries across nodes when it sees ResourceExhausted, and there is no structured signal to tell whether the whole fleet is saturated or just a subset of nodes.

Fix

  • Add Current() int64 and Limit() int64 methods to AdjustableSemaphore in packages/shared/pkg/utils/resizable_semaphore.go
  • Update all three ResourceExhausted error messages to include current=N, max=M

Example after fix:

"max number of running sandboxes on node reached: current=50, max=50, please retry"
"too many sandboxes starting on this node: current=10, max=10, please retry"
"too many sandboxes resuming on this node: current=10, max=10, please retry"

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading packages/shared/pkg/utils/resizable_semaphore.go, then inspect the three ResourceExhausted sites in sandboxes.go and utils.go. Add the semaphore count methods and update each error to report current and max values; done means all three messages include those counts.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
74/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.