In-flight runs die as sandbox_lost on any control-plane restart — runner should tolerate transient API outages
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 71
- Forks
- 64
- Avg merge
- 15h 38m
- Merged PRs (30d)
- 66
Description
Found dogfooding (#11/#13), three occurrences in one day. Any restart of the control plane while a sandbox run is in flight kills the run as sandbox_lost — and in a dev loop, restarts are constant and often implicit: a git merge/branch switch in the deployment's working tree touches files, tsx watch restarts the API/worker, the runner's next API call fails, the agent process dies, and reconcile marks the sandbox lost. Three architect runs died this way today (e.g. run_019f8aeb…, run_019f8b1f…), costing their full agent time.
Why it matters beyond dev
The same applies in production: an API deploy/restart should not kill every in-flight run. Sandboxes are already independent containers — the weak link is the runner treating a transient control-plane outage as fatal.
Ask
- Runner-side retry with backoff on control-plane calls (event emission, state upload, result posting) — survive outages of, say, up to 2–3 minutes before giving up. Buffer events locally in the interim.
- Reconcile grace: don't mark a sandbox lost while its container is still running and healthy (the container was alive in today's cases; it was the runner process inside that exited on API failure — distinguish "container gone" from "runner gave up").
- Optionally, on worker startup: re-adopt healthy running sandboxes instead of failing them.
Acceptance
- Restarting the API/worker mid-run (dev watch or deploy) does not fail an in-flight run; it completes and posts its result once the control plane is back.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the runner's control-plane calls and the reconcile path described in the issue, then reproduce an API or worker restart during an in-flight run. Trace event emission, state upload, result posting, and sandbox-loss handling. Done means the acceptance scenario completes successfully and posts its result once the control plane returns.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- api, backend, devops
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100