coder / coder/xum

Harden config persistence and orphan cleanup against disk exhaustion / config-load failure

Open
#4,197 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
2k
Forks
134
Avg merge
14h 57m
Merged PRs (30d)
307

Description

## Summary
During a September 9, 2026 incident in a Coder dogfood workspace, disk-full errors preceded detection of an incomplete Mux config and fallback to an empty registry. The UI appeared to have lost all projects/workspaces; the user restored the config.

Disk exhaustion is the leading trigger, not a proven explanation of the exact write failure. It is not yet established whether exhaustion occurred at the workspace volume, quota, or underlying shared host layer.

## Evidence reported from the workspace investigation
- At 16:39 and 16:43 UTC, the live Mux server log reported `ENOSPC: no space left on device` during metadata saves, status persistence, workspace-recency updates, and workflow lock-directory creation.
- At 16:43:41 UTC, Mux rejected the config and preserved it as `~/.mux/config.json.corrupt-1788972221254`.
- That backup was exactly 684,032 bytes; JSON parsing failed at end of file. Separate extension metadata also became invalid around the same time.
- The running Mux Node process had been alive since 08:04:37 UTC. The observed disappearance therefore did not require a server-process restart; a UI reload/reconnect may have looked like one.
- After recovery, the config parsed successfully and contained 31 projects. This does not establish that every historical session was recovered.
- Relevant server log: `~/.coder-modules/coder/mux/logs/mux.log`. In this instance, `~/.xum` resolves to `~/.mux`.

These findings were reported in the incident investigation; this issue does not include raw logs or config contents because they may contain sensitive data.

## Secondary risk: orphan cleanup after empty fallback
The code audit identified that `cleanupOrphanSessionDirs` removes session directories absent from the config when idle for more than 24 hours. If config loading fails and Mux substitutes an empty registry, otherwise valid sessions could consequently be treated as orphans on startup.

No actual orphan-session deletion was found in the retained server logs for this incident. This is a potential amplification path, not a confirmed loss mechanism here.

## Expected behavior / investigation
- Reproduce config persistence under `ENOSPC` and other write/flush/rename failures; identify how incomplete content reached the live config despite the apparent atomic-replacement write path.
- Preserve the last known-good config on failed saves, and surface a clear actionable storage/persistence error rather than silently presenting an empty installation.
- Fail closed for destructive orphan cleanup when config loading failed or the registry is an untrusted fallback. A failed load must not establish that sessions are orphaned.
- Add regression tests covering failed persistence, invalid/incomplete config recovery, and cleanup suppression after config-load failure.

## Unknowns
The exact writer/failing operation, affected build/commit, and exhausted storage layer still need confirmation. Another user reported a similar empty-config incident, but a shared cause has not been established. No PR is identified as the cause.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.