coder / coder/xum

Archive gating cannot see remote background bash jobs surviving an unclean restart (Coder stop policy)

Open
#3,944 2 comments 0 reactions 0 assignees View on GitHub
investigation
Dominant language
TypeScript
Stars
2k
Forks
134
Avg merge
14h 57m
Merged PRs (30d)
307

Description

### Context

PR #3940's crash-orphan guard makes model-driven archives fail closed when durable spawn records under `/tmp/mux-bashes/` show a background process that survived an unclean Xum restart. That layout is intentionally local-only: for SSH/Coder runtimes, `spawnProcess` writes `meta.json`/`exit_code` through the remote runtime (`runtime.tempDir()`), so the records live on the remote host.

### Problem (Codex review round 18, P2)

Condition: a dedicated Coder target uses the default `stop` archive policy, a **remote** background bash job survives an unclean Xum restart, and an orchestrator archives the workspace via `task_workspace_lifecycle`.

After the restart the in-memory `BackgroundProcessManager` map is empty and the local crash-orphan probe cannot see the remote records, so both archive gates report no background activity. The before-archive hook then stops the Coder workspace underneath the still-running remote job.

Non-crash flows are already covered (in-memory tracking + fresh status refresh), as are untrackable native terminals/editors under Coder `stop` (durable local markers).

### Suggested directions (design decision needed)

1. **Durable local admission registry for remote spawns**: write a session-dir record (mirroring `bashMonitorRegistryStore`) when spawning on a non-local runtime; remove it when the process is observed exited/terminated/cleaned. Leftover records after a restart fail the archive gate closed. Needs a reconciliation story so a job that exited during the outage does not permanently refuse model-driven archives (e.g. verify via remote `kill -0` when the workspace is reachable).
2. **Remote record probing at gate time**: exec a liveness check against the remote spawn layout during archive gating. Must not wake stopped Coder workspaces (`coder ssh --wait=yes` starts them) — probe only when the workspace is already running.

Either way, the refusal should route to user-mediated archive like the existing gates.

### References

- Review thread: PR #3940, round 18 (`src/node/services/backgroundProcessManager.ts`, `hasOrphanedRunningBackgroundProcesses` local-only scope comment)
- Local guard: `BackgroundProcessManager.hasOrphanedRunningBackgroundProcesses`, `localBgWorkspaceDir` (src/node/services/backgroundProcessExecutor.ts)
- Coder stop hook: `coderLifecycleHooks` / `coderWorkspaceArchiveBehavior`

---
_Generated with [`mux`](https://github.com/coder/mux) • Model: `anthropic:claude-fable-5` • Thinking: `xhigh`_

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.