cloudflare / cloudflare/sandbox-sdk
Mounting N buckets costs N × 7 sequential round trips; 3 mounts = 5.2s, ~40% of our cold start
- Dominant language
- TypeScript
- Stars
- 1.1k
- Forks
- 114
- Avg merge
- 22h 42m
- Merged PRs (30d)
- 14
Description
## Summary
Mounting a bucket costs ~1.6–1.8s, and because `MountOperationQueue` serializes all mount operations, mounting N buckets costs N × that. In our production sandboxes, mounting 3 buckets takes **5.2s — about 40% of a cold start**.
Most of that time is not the FUSE mount. Each `mountBucket()` performs **7 sequential round trips**, and only 2 of them are inherently per-mount.
Versions: observed on `0.13.0-next.*`; the same code path is on `main` (`0.12.9`).
## Where the time goes
`mountR2EgressBucket` (`packages/sandbox/src/storage-mount/operations/r2-egress-mount.ts`) awaits, in order:
| # | Call | Per-mount? |
|---|---|---|
| 1 | `files.writeFile(passwordFilePath)` | content is `` `${bucket}:x:x` `` — per binding, but regenerated at a fresh `randomUUID()` path every time |
| 2 | `mounts.chmodOwnerOnly(passwordFilePath)` | ↑ |
| 3 | `files.writeFile(headerFilePath)` | content is **always the constant `" Expect:\n"`** |
| 4 | `mounts.chmodOwnerOnly(headerFilePath)` | ↑ |
| 5 | `configureR2EgressOutbound(...)` | rebuilds params from the **whole** registry and overwrites the single `r2.internal` route table |
| 6 | `mounts.ensureDirectory(mountPath)` | ✅ genuinely per-mount |
| 7 | `executeS3FSMount(...)` | ✅ genuinely per-mount |
Steps 1–4 come from `s3fs/support-files.ts`, step 5 from `storage-mount/outbound/index.ts`.
A trivial container command in our sandboxes measures ~400ms round trip, which lines up with the observed per-mount cost:
```
08:29:23.387 bucket.mount /workspace/artifacts 1812ms
08:29:25.089 bucket.mount /workspace/assets 1651ms <- starts when the previous one ends
08:29:26.802 bucket.mount /workspace/references 1587ms <- same
```
So mounting 3 buckets is 21 sequential round trips, of which 6 are the actual mounts.
## Why this hits every multi-bucket user
Two design points compound here, and neither is avoidable from the caller side:
1. **One prefix per binding.** A binding can only be mounted once with a single prefix (`validateR2EgressMount` enforces it), so exposing two prefixes of the same bucket requires two bindings and therefore two mounts. We hit exactly this: two of our bindings point at the same bucket purely to mount `/projects//materials/` and `/references//` separately. The API shape forces the repetition.
2. **The queue then multiplies it.** To be clear: `MountOperationQueue` serializing is *correct* — `validateR2EgressMount` is a check-then-act over the shared registry, and step 5 is a whole-table overwrite of `r2.internal`. Sharding the queue per `mountPath` would break both invariants, so that is explicitly **not** what we are asking for. The issue is that the serialized section contains repeated global-scope work.
## What we are asking
Is there room to reduce the per-mount round trips, or to expose a batched mount? We are deliberately not proposing a design — the trade-offs are yours — but the shape of the redundancy looks like:
- Step 3/4 writes the same constant file to a new random path on every mount. (We do see why the random path exists: it lets each mount own and delete its own support file on unmount. A shared path would need that ownership rethought — but it is a 9-byte constant.)
- Step 5 rewrites the same global route table once per mount, where only the final state matters.
- Steps 6/7 are the only ones that must run per mount, and they are the cheap half.
A batched `mountBuckets([...])` would let validation and outbound configuration happen once and the per-mount work happen together, but we recognise the hard part there is failure semantics (partial success, registry and route-table convergence), not the happy path.
## Impact for us
Cold start (turn accepted → workload running) is p50 13.1s / p90 27.4s across our production sandboxes, and the three mounts are 5.2s of it — the single largest component. Warm starts, where the container is still alive and the mounts survive, are p50 5.2s, which is roughly the same figure with the mounting removed.
Happy to test a change against our workload if that helps.
Contributor guide
Assessment
This issue has not been assessed yet.