feat(volumes): pluggable distributed storage backend for high-performance cross-sandbox file sharing

Problem

The current persistent volume implementation backs every volume with a plain host directory and serves it to sandboxes over NFS v3 (packages/orchestrator/pkg/nfsproxy/). This works for sequential large-file access, but has hard limits for real workloads:

Workload	Current behaviour
Large-file sequential read/write	Acceptable, but adds NFS round-trip latency
Many small files (node_modules, Python packages, build trees)	Poor — every stat/open/readdir is a separate NFS RPC
High-concurrency writes from multiple sandboxes	No kernel-side cache; lock contention on the NFS server goroutine
Sandboxes on different orchestrator nodes sharing a volume	Not possible — volume lives on a single node's local filesystem

The last point is the most critical: as a fleet scales to multiple nodes, sandboxes scheduled on different nodes cannot share persistent volumes at all.

Proposed solution

Introduce a pluggable storage backend interface for persistent volumes, so the NFS chroot layer can be backed by any filesystem that satisfies a simple contract, while the default (local directory) is kept as-is.

Interface sketch

// packages/orchestrator/pkg/volumes/backend/backend.go
type Backend interface {
    // CreateVolume provisions storage for the given volume ID.
    CreateVolume(ctx context.Context, teamID, volumeID uuid.UUID) error

    // DeleteVolume removes all data for the volume.
    DeleteVolume(ctx context.Context, teamID, volumeID uuid.UUID) error

    // RootPath returns the host-side absolute path that the NFS chroot
    // (or future virtiofs passthrough) should expose to sandboxes.
    // The path must be accessible on every orchestrator node.
    RootPath(teamID, volumeID uuid.UUID) string
}

The existing volumes/volume_create.go + chrooted/builder.go path already calls os.MkdirAll(fullPath) — swapping fullPath to come from a Backend.RootPath() call is the only change needed in the hot path.

Target backends

JuiceFS (preferred first target)

• POSIX-compliant FUSE filesystem backed by object storage (S3, GCS, R2) + Redis/TiKV for metadata
• Single binary, no cluster to operate — easy to self-host
• Metadata latency for small files is ~1 ms (Redis) vs ~10 ms+ (NFS RPC over loopback)
• Mount once on the host at /mnt/juicefs/<team>/<vol>; sandbox chroot sees it as a normal directory
• Handles millions of small files well (Python envs, npm packages, build caches)

Ceph (CephFS)

• Kernel or FUSE client; fits teams already running Ceph
• Better for large-file, high-throughput workloads (ML checkpoints, video assets)
• Requires a Ceph cluster, so higher operational overhead

Generic mountpoint backend

• Simplest: operator pre-mounts any filesystem at a configured root (NFS, SMB, Lustre, Weka, etc.) and points the backend at it
• Zero code beyond reading the configured path

Configuration

# orchestrator environment / config
volumes:
  backend: juicefs          # "local" | "juicefs" | "cephfs" | "mountpoint"
  juicefs:
    meta_url: "redis://redis:6379/1"
    storage: s3
    bucket: s3://my-bucket/e2b-volumes
    access_key: ...
    secret_key: ...
  mountpoint:
    root: /mnt/shared-volumes   # pre-mounted by operator

Why this matters

• Cross-node volume sharing: sandboxes on any node can read/write the same volume — essential for stateful multi-step workflows (agent pipelines, CI builds, data processing)
• Small-file performance: JuiceFS with a Redis metadata store is 5–20× faster than NFS for workloads dominated by stat/open on many small files
• Durability: object-storage backends survive orchestrator node failure; local-directory volumes do not
• No API change: the POST /volumes + VolumeMounts API surface stays identical; the backend is an operator concern

Scope

• Define Backend interface in packages/orchestrator/pkg/volumes/backend/
• Refactor volumes/volume_create.go and chrooted/builder.go to consume Backend.RootPath() instead of constructing paths directly
• Implement LocalBackend (wraps current os.MkdirAll behaviour, default)
• Implement MountpointBackend (operator-managed pre-mount, zero deps)
• Document JuiceFS and CephFS as supported external backends with example configs
• Integration test: two sandboxes on the same node share a volume through a MountpointBackend backed by a tmpfs

Out of scope for this issue: virtiofs transport improvement, cross-team volume sharing.

References

• Current NFS proxy: packages/orchestrator/pkg/nfsproxy/
• Chroot builder: packages/orchestrator/pkg/chrooted/builder.go
• Volume create: packages/orchestrator/pkg/volumes/volume_create.go
• Config model: packages/orchestrator/pkg/cfg/model.go (PersistentVolumeMounts)
• JuiceFS architecture
• CephFS

Start by reading packages/orchestrator/pkg/volumes/volume_create.go, packages/orchestrator/pkg/chrooted/builder.go, packages/orchestrator/pkg/cfg/model.go, and the current NFS proxy under packages/orchestrator/pkg/nfsproxy/. Define the backend boundary and verify how volume paths and configuration flow through the existing code. Done means LocalBackend and MountpointBackend work, external backend configuration is documented, and the two-sandbox tmpfs integration test passes.