cloudflare / cloudflare/sandbox-sdk
Local bucket mounts fail for large files and after DO restarts (six bugs, fixes on a branch)
- Dominant language
- TypeScript
- Stars
- 1.1k
- Forks
- 114
- Avg merge
- 22h 42m
- Merged PRs (30d)
- 14
Description
We hit six bugs in the local-dev bucket mount path (`mountBucket(..., { localBucket: true })`) while running an agent workload against `wrangler dev` with local containers, where `/dev/fuse` is unavailable and local sync is the only way to get an archive in or out of a sandbox. Together they make local mounts unusable for anything larger than a few MB, and permanently broken after a Durable Object restart.
A complete fix with tests exists on a public branch, one commit on top of `main` (32e49a2):
- Branch: https://github.com/danieljvdm/sandbox-sdk/tree/fix/local-mount-sync-large-files
- Diff against main: https://github.com/cloudflare/sandbox-sdk/compare/main...danieljvdm:sandbox-sdk:fix/local-mount-sync-large-files
This repository limits opening pull requests to collaborators, so I am filing the analysis here instead. A maintainer is welcome to pull the branch directly, or invite the PR and I will open it immediately.
The same six fixes, applied as a patch over the published 0.12.4 bundle, are what we run locally today: a full end-to-end (baseline prep → task creation → agent turn → 13.8 MB checkpoint restore through the mount → second turn) passes with them and fails without.
**Environment:** `@cloudflare/sandbox` 0.12.4 (analysis and branch verified against `main`, 0.12.5), `SANDBOX_TRANSPORT=rpc`, `wrangler dev` local containers, `localBucket: true` mounts.
Each section below pairs one bug with what the branch does about it.
## Repro
An R2 bucket holding a 13.8 MB checkpoint archive, mounted for local dev:
```ts
await sandbox.mountBucket('WORKSPACE', '/workspace', { localBucket: true });
```
The initial R2 → container sync kills the RPC session:
```text
OperationInterruptedError: Sandbox operation files.writeFile was interrupted
while the runtime connection was closing (kind: session_disposed)
caused by: RPC session was shut down by disposing the main stub
```
With a ~1 GB object the same write fails earlier, in the Worker:
```text
RangeError: String is too long for a V8 string
at uint8ArrayToBase64 (packages/sandbox/src/local-mount-sync.ts)
```
Writing a large file *into* the mount fails the same way on its way back to R2. Calling `mountBucket()` again for the same bucket and path — which callers do because they cannot tell whether a warm sandbox already has the mount — throws `InvalidMountConfigError: Mount path already in use`. And after the Worker restarts (or the DO is evicted while the container idles out), every mount attempt fails with the `session_disposed` interruption above and never recovers.
## 1. A base64 write cannot exceed one control frame
`transferR2ObjectToContainer()` does `bucket.get()` → `arrayBuffer()` → `Buffer.toString('base64')` → `files.writeFile(..., { encoding: 'base64' })`.
On the rpc transport that payload rides a **single WebSocket frame**, and the container control server caps frames at 16 MiB (Bun's default `maxPayloadLength`). Base64 inflates content by 4/3, so anything over ~12 MB overruns the frame; the control socket closes, and capnweb surfaces it as `RPC session was shut down by disposing the main stub`, failing every in-flight and subsequent call on that session. This is the limit users hit first — long before the V8 string limit, which stops base64 working at all a few hundred MB later.
**Fix:** objects above 4 MiB stream straight into the container via `files.writeFileStream(path, obj.body, sessionId)` on the rpc transport — the same approach `doRestoreBackupLocal()` already uses for backup archives. 4 MiB keeps the base64 path well inside the frame budget. Below the threshold, and on the http and websocket transports (which have no stream write), the base64 path is unchanged.
## 2. Container → R2 reads the whole file as one base64 string
`uploadFileToR2()` reads the file with `readFile(..., { encoding: 'base64' })` and `put()`s the decoded bytes — the whole file in Worker memory, and past the V8 string limit no upload at all.
**Fix:** read with `readFileStream()` through the existing `streamFile()` SSE decoder and accumulate exactly 16 MiB parts. A file larger than one part goes up as an R2 multipart upload; smaller files keep the single `put()`. Two details worth a look while reviewing:
- Parts are sliced to an exact size (`takePart()`) rather than left chunk-aligned, because R2 rejects a multipart upload whose non-final parts differ in size.
- A failing part aborts the upload instead of leaving its uploaded parts behind in the bucket.
The post-upload `head()` snapshot refresh is unchanged, so echo suppression against the R2 poll loop still works.
## 3. Concurrent transfers share one RPC session
Both R2 → container loops (`fullSyncR2ToContainer()` and `pollR2ForChanges()`) run `Promise.all` batches of `SYNC_CONCURRENCY` transfers. A streamed transfer holds a stream export open on the shared capnweb session; when a sibling in the batch rejects, `Promise.all` abandons that export mid-flight, the session is torn down, and every other call on it fails — including the rest of the sync.
**Fix:** both loops transfer one object at a time. Error attribution also becomes unambiguous: a failure names the object it belongs to. If you would rather keep concurrency for objects below the streaming threshold, that is easy to add on top — the branch goes with sequential as the safe default.
## 4. Every inotify event uploads the whole file
`runContainerWatchLoop()` uploads on every `create`/`modify`/`move_to`. One large file being written into the mount (`tar -x`, `cp`) emits a burst of `modify` events, so that file is uploaded again and again, each time in a partial state.
**Fix:** a trailing 1.5 s per-path debounce, so the last event of a burst wins and each settled file is uploaded once. `delete`/`move_from` cancels a queued upload before deleting the object, so a queued upload cannot resurrect a deleted file. `stop()` clears pending timers and reports how many it dropped (`droppedUploads` in the stop log): once the watch stream is gone, nothing can tell whether those files finished being written. Uploads had no completion guarantee across `stop()` before this change either — the watch loop was abandoned mid-await.
## 5. The default session outlives the container that created it
`ensureDefaultSession()` returns the cached `defaultSession` on a bare id match. But `defaultSession` is restored from durable storage on cold start, while `containerGeneration` is memory-only and restarts at 0 — so a Durable Object evicted while its container was replaced comes back holding a session id the new container runtime never created. Every session-scoped call then fails with `OperationInterruptedError` (`session_disposed`), and because nothing invalidates the cached id, the sandbox never recovers. Mounts are simply where this shows up first: `LocalMountSyncManager` does all of its file I/O on the default session.
**Fix:** the cache carries the generation it was created against (`defaultSessionGeneration`, initially `-1`), and both the fast path and the invalidated-init retry require it to match `containerGeneration`. A session id restored from storage therefore costs exactly one re-initialization per DO instance, and against a still-live container that re-init is a no-op through the existing `SessionAlreadyExistsError` path.
## 6. Local mounts cannot be re-mounted
`mountBucketLocal()` throws `Mount path already in use` whenever `activeMounts` holds the path. But a local-sync mount is a plain directory, so `mountpoint -q` inside the container never reports it as mounted; the probe-then-mount pattern that works for FUSE mounts therefore re-mounts on every call and fails on the second one against a warm sandbox.
**Fix:** a repeat mount of the same bucket, prefix and `readOnly` returns successfully without starting a second sync loop — provided the existing mount belongs to the current container generation. If it belongs to an earlier one its sync manager holds a dead session, so it is stopped, dropped, and replaced with a fresh mount against the live container. Anything else — a different bucket, prefix or `readOnly`, or a mount of another type — still throws. `LocalSyncMountInfo` carries `prefix`, `readOnly` and `containerGeneration` for those comparisons, mirroring `R2BindingMountInfo`.
## What the branch contains
No public API change. 18 new unit tests (13 added to `tests/local-mount-sync.test.ts`, 4 in a new `tests/local-mount.test.ts`, 1 in `tests/sandbox.test.ts`) and a patch changeset. Each new test was checked to fail without its corresponding source change. `npm run check` passes, and `npm test -w @cloudflare/sandbox` is green at 52 files / 1097 tests. E2E was not run there (it needs the container image built), though the equivalent patch is exercised end-to-end in our own stack as described above.
Behaviour on the http and websocket transports is unchanged: `writeFileStream` is rpc-only, so those transports keep the base64 write and its size ceiling. Small files keep the single `put()`, conflicting mounts still throw, and the session change costs one idempotent `createSession` per DO instance when a session id was restored from storage.
Happy to split this into separate PRs (bug 5 is not mount-specific — it affects every consumer of the default session), rework any of it to your taste, or just leave the branch here for someone to cherry-pick.
Contributor guide
Assessment
This issue has not been assessed yet.