cloudflare / cloudflare/sandbox-sdk

createBackup() always fails for >=10MiB archives when interceptHttps is enabled: multipart part uploads use in-process fetch, which never trusts the injected CA

Open
#892 1 comment 0 reactions 1 assignee Claimed by @scuffi View on GitHub
bug
Dominant language
TypeScript
Stars
1.1k
Forks
114
Avg merge
22h 42m
Merged PRs (30d)
14

Description

## Summary

With `interceptHttps` enabled, `createBackup()` **always fails for archives at or above 10 MiB** with:

```
Multipart upload failed: self signed certificate in certificate chain
```

Backups below the threshold succeed, and `restoreBackup()` succeeds at any size. The split is exact and it is not about size itself — it is about which HTTP client the transfer path uses:

| Transfer path | Implementation | CA source | Intercepted HTTPS |
|---|---|---|---|
| `uploadArchive` (single-stream upload, `< BACKUP_MULTIPART_MIN_SIZE`) | `curl` child process | system bundle / `CURL_CA_BUNDLE` | works |
| `downloadArchive` (restore, incl. parallel parts) | `curl` child process | same | works |
| **`uploadPart` (multipart upload, `>= BACKUP_MULTIPART_MIN_SIZE`)** | **in-process Bun `fetch()`** | Bun's process-wide trust store | **always fails** |

`BACKUP_MULTIPART_MIN_SIZE` is 10 MiB and `multipart` defaults to `true`, so no configuration is needed to hit this — a workspace merely has to grow past 10 MiB.

Affects `0.12.x` and `0.13.0-next.*` (verified against `main` @ `20f9da4a` and `next` @ `bc0de65a`).

## Root cause

`trustRuntimeCert()` (`packages/sandbox-container/src/cert.ts`) appends the injected CA to the system bundle and sets `NODE_EXTRA_CA_CERTS` / `SSL_CERT_FILE` / `CURL_CA_BUNDLE` / … — but it is called from `server.ts` **after `Bun.serve()` has already started**:

```ts
// packages/sandbox-container/src/server.ts
logger.info('Container server started', { … });

if (process.env.SANDBOX_INTERCEPT_HTTPS === '1') {
await trustRuntimeCert();
}
```

Bun fixes its TLS trust store at process startup. Setting `NODE_EXTRA_CA_CERTS` afterwards therefore only affects **processes spawned later** — which is exactly why every `curl`-based transfer path works and only the in-process `fetch()` in `uploadPart` fails.

`uploadPart` is the container's only in-process HTTPS call to an external origin (`tunnel-manager` and `port-service` both fetch over plain HTTP on localhost), so the blast radius is precisely this one function.

## Minimal reproduction (no Cloudflare account needed)

`bun interception-ca-repro.ts` — requires `openssl` on PATH:

```ts
// Bun fixes its TLS trust store at process startup, so setting
// NODE_EXTRA_CA_CERTS at runtime (what trustRuntimeCert() does, after
// Bun.serve() is already up) never reaches in-process fetch(). Child processes
// launched afterwards do pick it up.
import { mkdtempSync, writeFileSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';

const dir = mkdtempSync(join(tmpdir(), 'ca-repro-'));
const sh = (cmd: string) => {
const r = Bun.spawnSync(['sh', '-c', cmd], { cwd: dir });
if (r.exitCode !== 0) throw new Error(`${cmd}\n${r.stderr.toString()}`);
};

// A private CA plus a leaf for localhost — stands in for the CA that Cloudflare
// injects at /etc/cloudflare/certs/cloudflare-containers-ca.crt.
writeFileSync(join(dir, 'ext.cnf'), 'subjectAltName=DNS:localhost\n');
sh('openssl req -x509 -newkey rsa:2048 -keyout ca.key -out ca.crt -days 1 -nodes -subj "/CN=Repro CA" 2>/dev/null');
sh('openssl req -newkey rsa:2048 -keyout srv.key -out srv.csr -nodes -subj "/CN=localhost" 2>/dev/null');
sh('openssl x509 -req -in srv.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out srv.crt -days 1 -extfile ext.cnf 2>/dev/null');

const caPath = join(dir, 'ca.crt');
const server = Bun.serve({
port: 0,
tls: { cert: Bun.file(join(dir, 'srv.crt')), key: Bun.file(join(dir, 'srv.key')) },
fetch: () => new Response('ok')
});
const url = `https://localhost:${server.port}/`;

const attempt = async (label: string, run: () => Promise) => {
try {
await run();
console.log(`${label}: OK`);
} catch (error) {
console.log(`${label}: FAIL — ${(error as Error).message}`);
}
};

// What the container does today: trustRuntimeCert() sets this after startup.
process.env.NODE_EXTRA_CA_CERTS = caPath;

await attempt('A in-process fetch(), NODE_EXTRA_CA_CERTS set at runtime', () => fetch(url));
await attempt('B in-process fetch() with tls: { ca } ', async () =>
await fetch(url, { tls: { ca: await Bun.file(caPath).text() } })
);
await attempt('C child process (curl), inherits the env ', async () => {
const proc = Bun.spawn(['curl', '-fsS', '-o', '/dev/null', url], {
env: { ...process.env, CURL_CA_BUNDLE: caPath },
stderr: 'pipe'
});
const code = await proc.exited;
if (code !== 0) throw new Error((await new Response(proc.stderr).text()).trim());
});

server.stop(true);
```

Output on Bun 1.3.8:

```
A in-process fetch(), NODE_EXTRA_CA_CERTS set at runtime: FAIL — unable to verify the first certificate
B in-process fetch() with tls: { ca } : OK
C child process (curl), inherits the env : OK
```

A is `uploadPart()`. C is `uploadArchive()` / `downloadArchive()`. B is the fix.

## End-to-end reproduction

```ts
import { Sandbox as SandboxBase, ContainerProxy, getSandbox } from '@cloudflare/sandbox';

export { ContainerProxy };

export class Sandbox extends SandboxBase {
enableInternet = true;
interceptHttps = true; // remove this line and the same backup succeeds
}

export default {
async fetch(request: Request, env: Env): Promise {
const sandbox = getSandbox(env.Sandbox, 'backup-repro');
await sandbox.exec('mkdir -p /workspace/big && dd if=/dev/urandom of=/workspace/big/blob.bin bs=1M count=12 status=none');
const backup = await sandbox.createBackup({ dir: '/workspace/big', ttl: 3600 });
return Response.json(backup);
}
};
```

`count=12` (>= 10 MiB) fails with `Multipart upload failed: self signed certificate in certificate chain`; `count=8` succeeds. Passing `multipart: false` also makes the 12 MiB case succeed, which is the workaround we are shipping meanwhile.

Note that `tests/e2e/backup-workflow.test.ts` does cover a 40 MB backup, but with `localBucket: true` and no `interceptHttps` sandbox in the e2e suite at all — so the multipart + interception combination is currently untested.

## Impact

For consumers that keep durable workspace state in backups, the failure is silent and cumulative: the archive is created, only the upload fails, so the checkpoint never advances while the container keeps serving. Every subsequent cold start restores the last checkpoint written before the workspace crossed 10 MiB. In our case a project silently lost a full day of work across five turns, because the session directory grows monotonically and never drops back below the threshold — once crossed, it is unrecoverable without intervention.

## Fix

PR: pass the injected CA per request in `uploadPart` via Bun's `tls: { ca }` fetch option, sourced from a `getRuntimeCertPem()` accessor that `trustRuntimeCert()` populates. That is the only place that needs it today, and it does not depend on startup ordering.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.