nfsproxy: hardcoded cacheLimit=1024 causes ESTALE errors on large npm/pip installs
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 1.6k
- Forks
- 438
- PR merge metrics
- No merged PRs in 30d
Description
Symptom
After mounting a persistent volume into a sandbox, large package installs (e.g. npm i -g @openai/codex) fail with:
npm error code UNKNOWN
npm error syscall write
npm error errno -116
npm error UNKNOWN: unknown error, write
errno -116 = ESTALE (Stale NFS file handle).
Root cause
packages/orchestrator/pkg/nfsproxy/proxy.go:25 hard-codes the NFS file-handle LRU size:
const cacheLimit = 1024
handler = helpers.NewCachingHandler(handler, cacheLimit)
CachingHandler maps uuid → (filesystem, path) via an LRU. When the cache fills, the oldest entry is evicted. If the client then sends an RPC using that evicted handle, FromHandle can no longer resolve it and returns NFSStatusStale (NFS status 70), which the Linux kernel translates to ESTALE:
// helpers/cachinghandler.go:100
return nil, []string{}, &nfs.NFSStatusError{NFSStatus: nfs.NFSStatusStale}
Amplifying factor: the 1024-slot LRU is shared across all sandboxes on the node — not per-sandbox. Under concurrent load, effective slots per sandbox are even fewer.
Why 1024 is not enough
@openai/codex fetches 6 platform-specific optional packages simultaneously (linux-x64/arm64, darwin-x64/arm64, win32-x64/arm64). npm's reify phase issues concurrent CREATE/RENAME/WRITE RPCs for each; every file and directory consumes a handle slot. 1024 is exhausted quickly.
Similar workloads that trigger this:
- Any large
npm iwith per-platform optional packages pip installof ML frameworks with many transitive dependencies- Extracting large archives into the volume
Proposed fix
Replace the hardcoded constant with an env-var-configurable value so operators can tune it without recompiling the orchestrator:
- Add
NFS_PROXY_CACHE_LIMIT intto the orchestrator config (pkg/cfg/model.go) withenvDefault:"16384" - Raise the default from 1024 → 16384 (each entry ≈150 bytes, 16384 entries ≈ 2.4 MB — negligible)
- Thread the value through
nfsproxy/cfg.Config→nfsproxy.NewProxy()
Longer-term: give each sandbox/mount its own CachingHandler instance to eliminate cross-sandbox LRU contention entirely.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with packages/orchestrator/pkg/nfsproxy/proxy.go, pkg/cfg/model.go, and helpers/cachinghandler.go to trace the cache-limit configuration into nfsproxy.NewProxy(). Make NFS_PROXY_CACHE_LIMIT configurable with a default of 16384, then verify the value reaches the shared CachingHandler and large concurrent installs no longer exhaust the 1024-entry limit.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, infrastructure
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 72/100