cloudflare / cloudflare/sandbox-sdk

Backup restore of large (>=10MB) archives is unrecoverable: parallel download script ends in `exit 1`, which kills the persistent backup session shell

Open
#884 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
1.1k
Forks
114
Avg merge
22h 42m
Merged PRs (30d)
14

Description

## Summary

When a workspace backup archive is `>= BACKUP_DOWNLOAD_PARALLEL_MIN_SIZE` (10 MB), `restoreBackup` takes the 8-way parallel `curl | dd` download path. That path builds a **single shell script that ends in `exit 1`** on any part failure, and it runs inside the **persistent backup session shell**. Because `exit` terminates that long-lived `bash` process, one failed part doesn't just fail the download — it kills the session (`Shell terminated unexpectedly … Session is dead`), which the restore layer surfaces as a non-recoverable error. After `RESTORE_MAX_ATTEMPTS` the caller's restore gate abandons the backup and resets the workspace, so a sandbox holding a large workspace can silently lose all its state the first time it sleeps and wakes.

The small-archive path (single `curl`) does **not** use `exit`; it returns curl's exit code and throws a normal error, so its failures are recoverable. The bug is specific to the parallel path.

Affects at least 0.12.3 through 0.12.8 (current latest).

## Where

`downloadBackupParallel` (dist `sandbox-*.js`) builds this for `expectedSize >= BACKUP_DOWNLOAD_PARALLEL_MIN_SIZE`:

```bash
rm -f
truncate -s
(set -o pipefail; curl -sSf ... -H 'Range: bytes=0-...' | dd of= oflag=seek_bytes seek=0 conv=notrunc 2>/dev/null) & J0=$!
# ... 8 parts ...
wait $J0; E0=$?
# ...
FAILED=$(( $E0 + $E1 + ... ))
if [ "$FAILED" -ne 0 ]; then rm -f ; exit 1; fi
```

This is executed via `execWithSession(script, backupSession, …)`. In the container runtime the session is a single long-lived shell:

```js
this.shell = Bun.spawn({ cmd: ["bash", "--norc"], stdin: "pipe", ... })
```

so the script's `exit 1` terminates the session shell itself. The runtime then throws `ShellTerminatedError` (`Shell terminated unexpectedly (exit code: 1). Session is dead and cannot execute further commands.`) and marks the session dead.

## Impact

- One failed download part → `exit 1` → dead session → the whole restore attempt fails.
- Each retry opens a fresh backup session, but if the failure is deterministic (see below) every attempt dies the same way, so retries are exhausted and the workspace is reset. Downstream consumers experience this as total, silent loss of the sandbox's persisted state.
- Observed as a cluster of restore failures all within ~4–6s of `workspace restore started`, each ending in `Session '' shell exited (exit code: 1)`.

## Two separate defects

1. **Recoverability (this issue's core):** using `exit` to signal part-failure inside a persistent session shell converts a retriable download error into a dead session. Signalling failure without terminating the shell (e.g. run the script in a subshell / `bash -c`, or end with `false` / a captured non-zero return instead of `exit`) would let the existing retry logic actually retry.

2. **Why a part fails in the first place (likely resource-bound):** in our deployment the failure is *deterministic* for large archives on small instances, which points at the parallelism itself — 8 concurrent `curl` processes plus a ~200 MB file materialising under a constrained memory/disk cgroup. If that's an OOM/resource ceiling, fixing (1) alone won't stop the workspace reset; the parallel path also needs to bound concurrency (or degrade to single-stream) relative to the instance's resources. We haven't isolated the exact trigger yet and would value maintainer insight here.

## Suggested fixes

- Don't use `exit` to fail the parallel script inside a session shell; make a part failure a recoverable non-zero result so retries work.
- Consider bounding parallel-download concurrency by available container memory, or falling back to single-stream on small instance types, so large restores don't OOM.

## Workaround

We're forcing the single-`curl` path by patching `BACKUP_DOWNLOAD_PARALLEL_MIN_SIZE` to `Infinity`. It restores correctly (at the cost of parallel throughput), which is itself evidence that the problem is specific to the parallel path.

Contributor guide

Open the contributing guide

Research direction

Trace downloadBackupParallel in dist sandbox-*.js through execWithSession and the Bun.spawn bash session. Reproduce a failed parallel part, then verify that the restore reports a recoverable failure, the persistent session remains usable, and retries can proceed; investigate the separate resource-bound concurrency concern as well.

Written by the indexing model from the issue text.

Assessment

Tech stack
bash, bun, typescript
Domain
backend, devtools, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.