theam / theam/facility

In-flight runs die as sandbox_lost on any control-plane restart — runner should tolerate transient API outages

Open
#35 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
TypeScript
Stars
71
Forks
64
Avg merge
15h 38m
Merged PRs (30d)
66

Description

Found dogfooding (#11/#13), three occurrences in one day. Any restart of the control plane while a sandbox run is in flight kills the run as sandbox_lost — and in a dev loop, restarts are constant and often implicit: a git merge/branch switch in the deployment's working tree touches files, tsx watch restarts the API/worker, the runner's next API call fails, the agent process dies, and reconcile marks the sandbox lost. Three architect runs died this way today (e.g. run_019f8aeb…, run_019f8b1f…), costing their full agent time.

Why it matters beyond dev

The same applies in production: an API deploy/restart should not kill every in-flight run. Sandboxes are already independent containers — the weak link is the runner treating a transient control-plane outage as fatal.

Ask

  1. Runner-side retry with backoff on control-plane calls (event emission, state upload, result posting) — survive outages of, say, up to 2–3 minutes before giving up. Buffer events locally in the interim.
  2. Reconcile grace: don't mark a sandbox lost while its container is still running and healthy (the container was alive in today's cases; it was the runner process inside that exited on API failure — distinguish "container gone" from "runner gave up").
  3. Optionally, on worker startup: re-adopt healthy running sandboxes instead of failing them.

Acceptance

  • Restarting the API/worker mid-run (dev watch or deploy) does not fail an in-flight run; it completes and posts its result once the control plane is back.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the runner's control-plane calls and the reconcile path described in the issue, then reproduce an API or worker restart during an in-flight run. Trace event emission, state upload, result posting, and sandbox-loss handling. Done means the acceptance scenario completes successfully and posts its result once the control plane returns.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
api, backend, devops
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.