[Bug]: Orchestration read model and per-thread client/VCS state grow unbounded over uptime
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 23k
- Forks
- 5.9k
- Avg merge
- 11h 14m
- Merged PRs (30d)
- 357
Description
Before submitting
- I searched existing issues and did not find a duplicate.
- I included enough detail to reproduce or investigate the problem.
Area
apps/server
Steps to reproduce
- Run the server (
t3 serve) and keep it up for an extended session (many hours), using several concurrent threads and creating/archiving/deleting threads over time. - Watch the server process RSS and CPU time (for example with
ps/htop), and watch the web UI responsiveness. - Compare a freshly started server against one that has been up for 12+ hours with the same workload.
Observed on a real server: the process reached roughly 1 GB RSS with hundreds of minutes of accumulated CPU time, and the web UI became progressively laggy the longer the server stayed up.
Expected behavior
Steady-state memory and per-event CPU should stay roughly flat over uptime. Command processing cost should not scale with the total number of threads ever created, and deleted threads should not keep costing work forever.
Actual behavior
The in-memory orchestration command read model stores threads and projects as arrays. Every domain event runs an O(N) linear .find() over the threads array and produces a full-array .map() copy through the projector, so per-event work and allocation scale with the total thread count. Deleted and archived threads are never removed from the model (only timestamped), so N grows monotonically for the life of the process and every subsequent event gets slower, increasing GC pressure.
Two client-side contributors compound this in long-lived browser tabs:
- Several per-thread stores (preview state, right panel, diff panel, ui state) accumulate one entry per thread ever visited and are never pruned on thread deletion; the ready-made cleanup functions exist but are not wired into the delete flow.
previewStateAtomiskeepAlive, so its atoms are retained for the tab lifetime. - The
VcsStatusBroadcasterper-cwd status cache is never pruned; it grows one entry per distinct cwd for the life of the server process.
Impact
Major degradation or frequent failure
Version or commit
main @ 2640e6dcf
Environment
Linux server (t3 serve), Node 24; reproduced with multiple concurrent threads over long uptime.
Workaround
Restarting the server clears the in-memory model and temporarily restores responsiveness, but the growth returns over time.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the apps/server entry point used by t3 serve, trace the orchestration read model and its event projector, then follow thread deletion into the named per-thread stores. Inspect previewStateAtom and the VcsStatusBroadcaster cache, and use a long-running multi-thread workload to confirm that memory, per-event work, and browser state remain bounded after threads are deleted or archived.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- node.js, typescript
- Domain
- backend, frontend, performance
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100