pingdotgg / pingdotgg/t3code

[Bug]: Orchestration read model and per-thread client/VCS state grow unbounded over uptime

Open
#4,178 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

🚧 In Progress bug
Dominant language
TypeScript
Stars
23k
Forks
5.9k
Avg merge
11h 14m
Merged PRs (30d)
357

Description

Before submitting
  • I searched existing issues and did not find a duplicate.
  • I included enough detail to reproduce or investigate the problem.
Area

apps/server

Steps to reproduce
  1. Run the server (t3 serve) and keep it up for an extended session (many hours), using several concurrent threads and creating/archiving/deleting threads over time.
  2. Watch the server process RSS and CPU time (for example with ps/htop), and watch the web UI responsiveness.
  3. Compare a freshly started server against one that has been up for 12+ hours with the same workload.

Observed on a real server: the process reached roughly 1 GB RSS with hundreds of minutes of accumulated CPU time, and the web UI became progressively laggy the longer the server stayed up.

Expected behavior

Steady-state memory and per-event CPU should stay roughly flat over uptime. Command processing cost should not scale with the total number of threads ever created, and deleted threads should not keep costing work forever.

Actual behavior

The in-memory orchestration command read model stores threads and projects as arrays. Every domain event runs an O(N) linear .find() over the threads array and produces a full-array .map() copy through the projector, so per-event work and allocation scale with the total thread count. Deleted and archived threads are never removed from the model (only timestamped), so N grows monotonically for the life of the process and every subsequent event gets slower, increasing GC pressure.

Two client-side contributors compound this in long-lived browser tabs:

  • Several per-thread stores (preview state, right panel, diff panel, ui state) accumulate one entry per thread ever visited and are never pruned on thread deletion; the ready-made cleanup functions exist but are not wired into the delete flow. previewStateAtom is keepAlive, so its atoms are retained for the tab lifetime.
  • The VcsStatusBroadcaster per-cwd status cache is never pruned; it grows one entry per distinct cwd for the life of the server process.
Impact

Major degradation or frequent failure

Version or commit

main @ 2640e6dcf

Environment

Linux server (t3 serve), Node 24; reproduced with multiple concurrent threads over long uptime.

Workaround

Restarting the server clears the in-memory model and temporarily restores responsiveness, but the growth returns over time.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the apps/server entry point used by t3 serve, trace the orchestration read model and its event projector, then follow thread deletion into the named per-thread stores. Inspect previewStateAtom and the VcsStatusBroadcaster cache, and use a long-running multi-thread workload to confirm that memory, per-event work, and browser state remain bounded after threads are deleted or archived.

Written by the indexing model from the issue text.

Assessment

Tech stack
node.js, typescript
Domain
backend, frontend, performance
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.