Background service exhausts V8 heap and aborts after ~1-2 days of continuous uptime under sustained session load
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 23k
- Forks
- 5.9k
- Avg merge
- 11h 14m
- Merged PRs (30d)
- 357
Description
Summary
The background service (t3 serve, runtime 0.0.42) exhausts the V8 heap and
aborts after roughly a day to two days of continuous uptime, under sustained
agent-session load. It has happened twice in four days on the same host, with an
identical stack. Restart=always recovers it in ~25 seconds, but the restart
SIGKILLs everything in the cgroup, including in-flight provider sessions.
Occurrences
crash 1 2026-09-18 05:48:00Z uptime 48.0h cgroup memory peak 18.4 G
crash 2 2026-09-19 11:35:36Z uptime 29.8h cgroup memory peak 15.4 G
Both SIGABRT, both V8 FatalProcessOutOfMemory on the main thread:
node!v8::internal::V8::FatalProcessOutOfMemory
node!v8::internal::Heap::FatalProcessOutOfMemory
node!v8::internal::Heap::CollectGarbage
node!v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath
No --max-old-space-size and no NODE_OPTIONS are set, so the server runs at
V8's internal default ceiling.
RSS growth on the current process, for shape: 230 MB after 90 minutes of
uptime. Small at start, fatal after a day-plus.
Workload
A multi-agent orchestration host: many long-lived provider sessions, frequent
thread/turn churn, and git operations against numerous worktrees. The second
window consumed 13h57m of CPU over 29.8h wall-clock, so the process is busy
rather than idle.
The projection database (state.sqlite) held ~16.7k orchestration events and
~430 live session bindings at the time of the second crash, if session or
projection state is a plausible retention site.
On the interval
48.0h then 29.8h. I am not claiming acceleration — two points cannot show a
trend, and the shorter window also carried heavier load, which fits the same
data. What the data does support is that it is reproducible under sustained use
and is not a one-off.
Impact
Recovery is automatic and fast, but the cgroup teardown is not graceful. The two
crashes killed, respectively:
crash 1: 2x node children, 1x git
crash 2: 1x claude (a provider session), 1x bash, 1x cat
A provider session killed mid-run leaves no completion record, so from the
orchestrator's side the work simply stops. A git process killed mid-write is a
worse case — a wedged index or stale lock in the affected worktree.
Environment
runtime 0.0.42 (SEA binary, ./t3 __service-launcher)
t3code-bin 0.0.38-1 (Arch)
OS Arch Linux, kernel 7.1.9, systemd 261
host 8-core / 16-thread, 29 GiB RAM, zram swap
service user unit, Restart=always, StartLimitBurst=5/300s
heap cap none set
What would help
Happy to gather more from this host if it is useful — I can run the server with
--max-old-space-size set and --heapsnapshot-near-heap-limit=1 to capture a
snapshot at the ceiling rather than losing the process, if that would be worth
more than the figures above. Say what would be most useful and I will collect it
on the next occurrence rather than guess.
Not asking for
No workaround needed on my side — automatic restart is holding. Filing because
a long-running service reaching V8's ceiling under normal use looks like
something you would want to know about, and no existing issue covers it.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the t3 serve service and ./t3 __service-launcher entry points, then reproduce sustained session load while collecting the proposed V8 heap snapshot with --heapsnapshot-near-heap-limit=1. Inspect retained objects related to the ~16.7k state.sqlite orchestration events and ~430 live session bindings. Done requires identifying the retention site and demonstrating that long-running service use no longer reaches the V8 ceiling.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- node.js, typescript
- Domain
- backend, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 45/100