Optimize Firecracker sandbox startup by reducing mount namespace copies in the orchestrator

Background

We are planning to use E2B infra for large-scale sandbox workloads, including post-training reinforcement learning for large language models and commercial AI agent workloads. We expect to run a large number of sandboxes on high-core-count machines.

However, we found that under large-scale and high-concurrency workloads, sandbox startup latency becomes very high. It is far beyond Firecracker's expected sub-100 ms startup time. With 100 concurrent sandbox creation requests, creating a single sandbox can take more than 1 second in the orchestrator Firecracker startup path.

We investigated this issue and found that the high latency is mainly caused by lock contention during mount namespace operations.

Specifically, in packages/orchestrator/pkg/sandbox/fc/process.go, a large amount of time is spent in the Firecracker startup/configure path, especially around:

err = socket.Wait(startCtx, p.firecrackerSocketPath)

This is related to how the Firecracker environment is prepared before the Firecracker process becomes ready.

The current flow contains mount namespace related operations such as:

unshare -m
ip netns exec ns-xxx firecracker ...

Both commands can copy the mount tree and eventually reach copy_mnt_ns in the Linux kernel file fs/namespace.c. When the mount tree is copied, the kernel takes the namespace lock:

static inline void namespace_lock(void)
{
    down_write(&namespace_sem);
}

With many concurrent sandbox creation requests, multiple threads contend for this global lock, which makes sandbox startup latency increase sharply.

In addition:

1. Copying a mount tree requires traversing the mount tree, which is expensive when the host mount tree is large.
2. The previous flow performs more than one mount namespace related operation in the Firecracker startup path, including mount tree copies and mount namespace release. These operations can all touch the same namespace lock path.

We also ran a benchmark after creating 1000 sandboxes and then destroying them.

The environment is:

Kernel: 6.6.0-135.0.0.113.oe2403sp3.x86_64
CPU: AMD EPYC 9755
Cores: 512
Preparation: create 1000 sandboxes, then destroy them

In the table below, socket.Wait is included in the total orchestrator startup/configure time. It is the key step inside the measured orchestrator path, not a separate end-to-end measurement.

Concurrency	Before total	Before socket.Wait	After total	After socket.Wait
1	100.000 ms	62.167 ms	44.000 ms	12.636 ms
16	595.812 ms	557.007 ms	52.125 ms	21.681 ms
50	982.320 ms	900.674 ms	46.120 ms	22.734 ms
100	1063.110 ms	998.058 ms	46.270 ms	21.434 ms

The current path degrades significantly as concurrency increases. The total orchestrator startup/configure time grows from 100 ms at concurrency 1 to 1063.110 ms at concurrency 100. The key socket.Wait step grows from 62.167 ms to 998.058 ms.

After the optimization, the total time stays around 44-52 ms, and socket.Wait stays around 12-23 ms.

For these reasons, creating the Firecracker process has very large latency under high concurrency, and most of the latency comes from mount namespace related work in the startup path.

Proposed Optimization

We recommend changing the sandbox creation flow.

1. Move mount tree copy work out of the hot path

Before serving sandbox creation requests, the orchestrator can prepare fresh mount namespaces in advance.

Later, when creating each sandbox, the startup path can use a prepared mount namespace instead of copying the full host mount tree during the request. This moves the expensive mount tree copy work out of the latency-sensitive path.

2. Optimize the Firecracker process startup flow

Replace the previous unshare -m and ip netns exec ns-xxx firecracker ... flow with explicit setns system calls.

For example, Firecracker can be started through a small helper that:

1. enters the prepared mount namespace;
2. enters the target network namespace;
3. performs only the minimal per-sandbox mount/remount work;
4. execs the Firecracker process.

This reduces the amount of mount namespace work performed during sandbox startup and avoids repeatedly copying the host mount tree in the request hot path.

To keep isolation conservative, each sandbox can still receive a fresh mount namespace. The prepared mount namespace should be destroyed when the sandbox is cleaned up, instead of being reused across different sandboxes.

Benefits

This has several benefits:

1. It reduces sandbox creation time. In our benchmark, at 100 concurrency, the orchestrator Firecracker startup/configure path is reduced from 1063.110 ms to 46.270 ms, and the key socket.Wait(startCtx, p.firecrackerSocketPath) step is reduced from 998.058 ms to 21.434 ms. The optimized path also stays around 44-52 ms from 1 to 100 concurrency, instead of degrading sharply as concurrency increases.

2. It improves security. If a user compromises the Firecracker VM, they may be able to see the host mount tree and learn the host filesystem structure, which could enable further attacks. With a prepared fresh mount namespace, the attack surface is reduced, and mount state does not need to be reused across unrelated sandboxes.

Start in packages/orchestrator/pkg/sandbox/fc/process.go and trace the Firecracker startup path around socket.Wait(startCtx, p.firecrackerSocketPath), including the current unshare and ip netns exec flow. Review how mount and network namespaces are prepared and cleaned up, then benchmark concurrent sandbox creation to verify that startup remains near the reported 44-52 ms range and socket.Wait no longer degrades sharply.