cloudflare / cloudflare/sandbox-sdk

Session teardown kills all startProcess-managed process trees while the container keeps serving

Open
#891 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
1.1k
Forks
114
Avg merge
22h 42m
Merged PRs (30d)
14

Description

## Summary

On `@cloudflare/sandbox` 0.12.4 in production, we repeatedly observe a teardown event that kills **every process running under the sandbox's default session** — everything launched via `startProcess` and `exec`, including their detached-looking children — while the container VM itself, `/container-server/sandbox` (pid 65) and its `node_executor` processes all survive. The sandbox then keeps serving `exec` normally within ~1–2 seconds, attached to a half-dead container.

## What we measured (forensics from inside the container)

- Between two port probes ~20s apart, every workspace process died simultaneously: our agent engine (`startProcess`), a watchdog process (`startProcess`), and two supervisor trees. `ps -eo pid,etimes` afterwards shows the control server continuous since boot while every other process restarted.
- **Not the kernel**: `dmesg` is clean (no OOM lines), ~80% of memory free at the moment of death, `oom_score_adj -900` applied to the main victim.
- **Not our code**: we never call `killProcess`/`killAllProcesses`; the process registry records the victims as `status: "error"` (unexpected exit), not `"killed"`.
- The VM does not restart (dmesg uptime continuous; disk untouched; git history intact).
- From the Durable Object side, the moment coincides with `ReadableStream received over RPC disconnected prematurely` on a long-lived `containerFetch` SSE, and occasionally `"Container suddenly disconnected, try again"` on an in-flight request.
- Frequency: across 33 production build sessions in a 72h window, 29 (88%) hit at least one such event. It correlates strongly (~81% of incidents) with windows where a **second container** (a build sandbox on the same account/DO namespace) is being created/destroyed, though we have not proven causation.

## Why it hurts

Long-lived processes started with `startProcess` are the SDK's stated way to run persistent services. If some platform/runtime path (a canceled stop? a session re-init? graceful-shutdown cleanup that doesn't complete?) can kill the whole session process tree while leaving the container serving, every SDK consumer's long-running processes silently die with no callback, no registry "killed" status, and no signal to the DO other than broken streams.

Our mitigation (works, feels wrong): launching our supervisors via `setsid -f` so they live outside the session's process groups — they now survive these events and respawn our engine in milliseconds.

## Questions

1. What sends the signal? Is there a code path (platform stop initiated then canceled, session eviction, shutdown handler) that kills session process groups while the container-server continues serving?
2. If this is graceful-shutdown cleanup: is a stop-then-continue sequence expected to leave the server running with all managed processes dead?
3. Is there a supported way to mark a `startProcess` process as "survives session teardown"?

## Environment

- `@cloudflare/sandbox` 0.12.4 (client and image `cloudflare/sandbox:0.12.4`), Workers + Durable Objects + Containers (instance_type standard-2), production.
- Happy to share timestamps/account details privately for correlation with platform logs.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the teardown around startProcess and exec while monitoring the process registry, /container-server, node_executor processes, and containerFetch streams. Correlate the event with creation or destruction of a second container and determine what signal or lifecycle path kills the managed trees. Done means identifying the responsible path and defining or fixing the expected teardown behavior, including process status and notification.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, typescript
Domain
backend, distributed-systems, infrastructure
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.