e2b-dev / e2b-dev/runtime

nfsproxy: hardcoded cacheLimit=1024 causes ESTALE errors on large npm/pip installs

Open
#3,548 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
1.6k
Forks
438
PR merge metrics
No merged PRs in 30d

Description

Symptom

After mounting a persistent volume into a sandbox, large package installs (e.g. npm i -g @openai/codex) fail with:

npm error code UNKNOWN
npm error syscall write
npm error errno -116
npm error UNKNOWN: unknown error, write

errno -116 = ESTALE (Stale NFS file handle).

Root cause

packages/orchestrator/pkg/nfsproxy/proxy.go:25 hard-codes the NFS file-handle LRU size:

const cacheLimit = 1024
handler = helpers.NewCachingHandler(handler, cacheLimit)

CachingHandler maps uuid → (filesystem, path) via an LRU. When the cache fills, the oldest entry is evicted. If the client then sends an RPC using that evicted handle, FromHandle can no longer resolve it and returns NFSStatusStale (NFS status 70), which the Linux kernel translates to ESTALE:

// helpers/cachinghandler.go:100
return nil, []string{}, &nfs.NFSStatusError{NFSStatus: nfs.NFSStatusStale}

Amplifying factor: the 1024-slot LRU is shared across all sandboxes on the node — not per-sandbox. Under concurrent load, effective slots per sandbox are even fewer.

Why 1024 is not enough

@openai/codex fetches 6 platform-specific optional packages simultaneously (linux-x64/arm64, darwin-x64/arm64, win32-x64/arm64). npm's reify phase issues concurrent CREATE/RENAME/WRITE RPCs for each; every file and directory consumes a handle slot. 1024 is exhausted quickly.

Similar workloads that trigger this:

  • Any large npm i with per-platform optional packages
  • pip install of ML frameworks with many transitive dependencies
  • Extracting large archives into the volume

Proposed fix

Replace the hardcoded constant with an env-var-configurable value so operators can tune it without recompiling the orchestrator:

  • Add NFS_PROXY_CACHE_LIMIT int to the orchestrator config (pkg/cfg/model.go) with envDefault:"16384"
  • Raise the default from 1024 → 16384 (each entry ≈150 bytes, 16384 entries ≈ 2.4 MB — negligible)
  • Thread the value through nfsproxy/cfg.Confignfsproxy.NewProxy()

Longer-term: give each sandbox/mount its own CachingHandler instance to eliminate cross-sandbox LRU contention entirely.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with packages/orchestrator/pkg/nfsproxy/proxy.go, pkg/cfg/model.go, and helpers/cachinghandler.go to trace the cache-limit configuration into nfsproxy.NewProxy(). Make NFS_PROXY_CACHE_LIMIT configurable with a default of 16384, then verify the value reaches the shared CachingHandler and large concurrent installs no longer exhaust the 1024-entry limit.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, infrastructure
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.