[Bug]: systemd-oomd kills the entire T3 Code scope when an agent-spawned child workload exhausts memory (Linux)
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 23k
- Forks
- 5.9k
- Avg merge
- 11h 14m
- Merged PRs (30d)
- 357
Description
Summary
Everything T3 Code spawns runs in the single systemd scope the desktop app was launched into (app-t3code-<pid>.scope): the embedded server, agent (provider) processes, terminals, and all of their descendants (emulators, Gradle daemons, dev servers). When an agent's child workload exhausts memory, systemd-oomd kills the whole scope: the main Electron process, the server, and every other agent session die along with the actual memory hog.
This is a containment/blast-radius issue, distinct from the renderer memory work in #5148 and the backend hydration fixes in #5147: even with both of those merged, a single runaway child process tree still takes down all of T3.
Environment
- T3 Code v0.0.31 (desktop AppImage), Linux
- Fedora Linux 44 (KDE Plasma), kernel 7.0.12-201.fc44.x86_64
- 31 GiB RAM, 12 CPUs, systemd-oomd enabled (Fedora default)
Incident
During an agent session doing Android work (emulator + Gradle build), T3's scope grew until sustained memory pressure exceeded the user-slice oomd threshold, and systemd-oomd killed the entire scope, all 48 processes:
Aug 02 17:24:36 fedora systemd-oomd[675]: Considered 80 cgroups for killing, top candidates were:
Aug 02 17:24:36 fedora systemd-oomd[675]: Path: /user.slice/user-1000.slice/user@1000.service/app.slice/app-t3code-1223476.scope
Aug 02 17:24:36 fedora systemd-oomd[675]: Pressure: Avg10: 87.57, Avg60: 72.12, Avg300: 30.16, Total: 1min 54s
Aug 02 17:24:36 fedora systemd-oomd[675]: Current Memory Usage: 11.3G
Aug 02 17:24:36 fedora systemd-oomd[675]: Killed /user.slice/user-1000.slice/user@1000.service/app.slice/app-t3code-1223476.scope due to memory pressure for /user.slice/user-1000.slice/user@1000.service being 86.07% > 80.00% for > 20s with reclaim activity
Aug 02 17:24:36 fedora systemd[1659]: app-t3code-1223476.scope: systemd-oomd killed 48 process(es) in this unit.
Aug 02 17:24:38 fedora systemd[1659]: app-t3code-1223476.scope: Failed with result 'oom-kill'.
The failed unit's retained accounting confirms the totals: MemoryPeak=12198318080 (~11.4 GiB), MemorySwapPeak=928509952 (~885 MiB), Result=oom-kill.
Per-process breakdown (reproduced ~40 minutes later)
After relaunching T3 and resuming the same agent session, the identical pattern rebuilt. A live sample of the new scope (app-t3code-2525611.scope, ~11 GB total at 18:04) shows exactly where the memory is:
| RSS | Process | Parent |
|---|---|---|
| 3.8 GB | java (Gradle daemon) |
reparented to user manager, still in T3's scope |
| 3.5 GB | qemu-system-x86 (Android emulator) |
agent (claude) process |
| 1.6 GB | java (Gradle worker) |
Gradle daemon |
| 415 MB | claude (agent #1) |
T3 server |
| 321 MB | claude (agent #2) |
T3 server |
| 317 MB | t3code (embedded server) |
T3 main process |
| 237 MB | t3code (Electron main) |
n/a |
T3's own processes account for well under 1 GB; ~8.9 GB is the emulator/Gradle subtree. (Renderer/GPU processes live in a separate desktop-launcher unit that stayed under 800 MB.) But because oomd's unit of accounting and its kill unit is the cgroup, T3's scope is always the top kill candidate whenever any child workload misbehaves, and the kill destroys the UI, the server, and unrelated agent sessions.
Note the Gradle daemon: its original parent exited, it reparented to the user manager, but it stays in T3's cgroup. Long-lived daemons spawned by agent tasks keep counting against T3's scope even after the task that started them is gone.
Why this belongs in T3
- T3 spawns these workloads via
ChildProcessSpawner(apps/server/src/processRunner.ts,apps/server/src/process/externalLauncher.ts), which inherits the parent cgroup; there is no cgroup separation anywhere in the spawn path. - T3 already attributes this memory correctly: the resource telemetry (
apps/server/src/resourceTelemetry/,diagnostics/ProcessResourceMonitor.ts) tracks per-process RSS withprovider-root/terminal-rootcategories. But nothing acts on it, and the OS can't act on anything smaller than the whole scope. - There is already precedent for systemd integration in the codebase:
apps/server/src/cli/service.tsmanages the persistent-server unit on "Linux with systemd".
Proposed fix (Linux-only, tightly scoped)
Spawn each agent/terminal child tree into its own transient systemd scope (systemd-run --user --scope --collect, or StartTransientUnit over the user D-Bus session), e.g. t3code-task-<id>.scope. Effects:
- systemd-oomd accounts each task subtree separately and kills only the runaway scope; T3's UI, server, and other sessions survive.
- T3 can observe the scope's
oom-killresult and surface "this task was killed for memory" to the user instead of the whole app dying. - Orphaned daemons (like the Gradle daemon above) stay attributed to their task's scope instead of T3's.
- Optionally, a configurable
MemoryHigh=per task scope to throttle before pressure builds (off by default, no behavior change). - Graceful fallback: if there is no user systemd manager (non-systemd distros, containers), spawn exactly as today.
Related prior work: #5148 (renderer memory containment, open), #5147 (backend event-replay OOMs, merged), #2042 / #2359 (leaked child processes, merged). None of these isolate child workloads from T3's own cgroup. The closest prior issue is #1591 (T3's web server killed when a sibling process in a shared tmux scope OOMed, closed with "supervision is the recovery boundary"). This issue is the inverse and is T3's to fix: here T3 itself composes the kill unit, by placing its UI, server, and unrelated agent sessions in the same cgroup as arbitrary heavy child workloads. Terminal emulators solved this exact problem the same way this issue proposes: VTE/GNOME Terminal spawn each tab's shell into its own transient systemd scope so a runaway tab doesn't take down the terminal.
Per CONTRIBUTING's "Issues First": happy to send a small Linux-only PR implementing the transient-scope spawn with fallback, if there's interest in this direction.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with ChildProcessSpawner in apps/server/src/processRunner.ts and apps/server/src/process/externalLauncher.ts, then review the systemd integration in apps/server/src/cli/service.ts. Trace the Linux spawn path and fallback behavior before choosing between systemd-run and the user D-Bus API. Done means agent and terminal trees use separate transient scopes where available, with unchanged fallback behavior elsewhere.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- linux, typescript
- Domain
- backend, infrastructure, operating-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 48/100