pingdotgg / pingdotgg/t3code

Background service exhausts V8 heap and aborts after ~1-2 days of continuous uptime under sustained session load

Open
#12,584 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

accepted bug via-triage
Dominant language
TypeScript
Stars
23k
Forks
5.9k
Avg merge
11h 14m
Merged PRs (30d)
357

Description

Summary

The background service (t3 serve, runtime 0.0.42) exhausts the V8 heap and
aborts after roughly a day to two days of continuous uptime, under sustained
agent-session load. It has happened twice in four days on the same host, with an
identical stack. Restart=always recovers it in ~25 seconds, but the restart
SIGKILLs everything in the cgroup, including in-flight provider sessions.

Occurrences

crash 1   2026-09-18 05:48:00Z   uptime 48.0h   cgroup memory peak 18.4 G
crash 2   2026-09-19 11:35:36Z   uptime 29.8h   cgroup memory peak 15.4 G

Both SIGABRT, both V8 FatalProcessOutOfMemory on the main thread:

node!v8::internal::V8::FatalProcessOutOfMemory
node!v8::internal::Heap::FatalProcessOutOfMemory
node!v8::internal::Heap::CollectGarbage
node!v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath

No --max-old-space-size and no NODE_OPTIONS are set, so the server runs at
V8's internal default ceiling.

RSS growth on the current process, for shape: 230 MB after 90 minutes of
uptime. Small at start, fatal after a day-plus.

Workload

A multi-agent orchestration host: many long-lived provider sessions, frequent
thread/turn churn, and git operations against numerous worktrees. The second
window consumed 13h57m of CPU over 29.8h wall-clock, so the process is busy
rather than idle.

The projection database (state.sqlite) held ~16.7k orchestration events and
~430 live session bindings at the time of the second crash, if session or
projection state is a plausible retention site.

On the interval

48.0h then 29.8h. I am not claiming acceleration — two points cannot show a
trend, and the shorter window also carried heavier load, which fits the same
data. What the data does support is that it is reproducible under sustained use
and is not a one-off.

Impact

Recovery is automatic and fast, but the cgroup teardown is not graceful. The two
crashes killed, respectively:

crash 1:  2x node children, 1x git
crash 2:  1x claude (a provider session), 1x bash, 1x cat

A provider session killed mid-run leaves no completion record, so from the
orchestrator's side the work simply stops. A git process killed mid-write is a
worse case — a wedged index or stale lock in the affected worktree.

Environment

runtime          0.0.42 (SEA binary, ./t3 __service-launcher)
t3code-bin       0.0.38-1 (Arch)
OS               Arch Linux, kernel 7.1.9, systemd 261
host             8-core / 16-thread, 29 GiB RAM, zram swap
service          user unit, Restart=always, StartLimitBurst=5/300s
heap cap         none set

What would help

Happy to gather more from this host if it is useful — I can run the server with
--max-old-space-size set and --heapsnapshot-near-heap-limit=1 to capture a
snapshot at the ceiling rather than losing the process, if that would be worth
more than the figures above. Say what would be most useful and I will collect it
on the next occurrence rather than guess.

Not asking for

No workaround needed on my side — automatic restart is holding. Filing because
a long-running service reaching V8's ceiling under normal use looks like
something you would want to know about, and no existing issue covers it.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the t3 serve service and ./t3 __service-launcher entry points, then reproduce sustained session load while collecting the proposed V8 heap snapshot with --heapsnapshot-near-heap-limit=1. Inspect retained objects related to the ~16.7k state.sqlite orchestration events and ~430 live session bindings. Done requires identifying the retention site and demonstrating that long-running service use no longer reaches the V8 ceiling.

Written by the indexing model from the issue text.

Assessment

Tech stack
node.js, typescript
Domain
backend, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.