[Bug]: Closing any terminal hangs forever while another terminal emits output rapidly — new terminals can't open until the client reconnects

Disclaimer: This report was written by an AI (Claude Fable 5) investigating the issue on our server; it was reviewed and submitted by a human.

Before submitting

• I searched existing issues and did not find a duplicate.
• I included enough detail to reproduce or investigate the problem.

Area

apps/server

Steps to reproduce

1. In a project thread, open terminal A and run a redraw-heavy loop (same generator as #7991):

   while true; do printf '\r%s' "$(date +%s%N)"; done
   

2. Let it run 1–2 minutes.
3. Open terminal B in the same thread, then close B's tab.
4. Click "New Terminal".

Expected behavior

• Closing B completes in well under a second (B produced almost no output).
• A new terminal opens normally.

Actual behavior

• The terminal.close RPC for B never completes (we observed hangs of 24 s, 14 min, and 36 min — each ended only when the fiber was interrupted by a client reconnect, never by finishing).
• The "New Terminal" button stops responding: terminal.open/attach waits on the thread lock still held by the hung close. After a window reload interrupts the close, the same open completes in ~120 ms.
• Closing A itself hangs the same way.

Impact

Frequent failure — terminals in a thread become unmanageable (can't close, can't open new ones) whenever any terminal in it runs a redraw-heavy TUI or spinner. Additionally the persist worker rewrites the ever-growing history file in a tight loop; we measured the file cycling 131 MB → 0 → 131 MB every ~2 s (~65 MB/s sustained disk writes for hours).

Version or commit

t3 0.0.34 (server, t3 serve on Linux). The relevant logic is unchanged on current main (packages/shared/src/KeyedCoalescingWorker.ts).

Environment

Ubuntu 24.04, kernel 6.8.0-138-generic, x86_64; client: T3 desktop app on Windows connecting over Tailscale Serve.

Logs or stack traces

# server.trace.ndjson — close hangs, open blocks behind it, then succeeds instantly after interrupt
12:11:44.920 ws.rpc.terminal.close   dur=24147.7ms  Interrupted
12:11:44.920 terminal.closeSession   dur=24147.6ms  Interrupted
12:11:44.963 terminal.persistHistory dur=24104.5ms  Interrupted
12:11:44.963 terminal.flushPersist   dur=24104.3ms  Interrupted
12:11:45.849 ws.rpc.terminal.attach  dur=23218.2ms  Interrupted   <- new terminal, blocked
12:12:09.179 terminal.openLocked     dur=122.6ms    Success       <- same open, right after interrupt

# earlier the same day
11:08:39 ws.rpc.terminal.close      dur=2178164.7ms Interrupted   <- 36 minutes

# persist worker runs back-to-back for hours on one key (7,666 spans, avg 720ms)
12:50:34 terminal.persistHistoryWorker dur=1147ms Success
12:50:35 terminal.persistHistoryWorker dur=916ms  Success
12:50:36 terminal.persistHistoryWorker dur=690ms  Success
...

Notes for triage

KeyedCoalescingWorker has a single consumer fiber, and processKey tail-recurses as long as latestByKey holds a newer value for the current key. A terminal emitting faster than one process() cycle (≥500 ms debounce + full history-file rewrite, which grows per #7991) therefore keeps the fiber captive on that key forever: TxQueue.take is never reached again, every other key starves, and any drainKey retries indefinitely. terminal.close calls flushPersist (= drainKey) while holding the per-thread lock, so terminal.open in that thread blocks too. #7991 aggravates this (each cycle gets slower as the file grows) but is a separate defect — this hang reproduces with a bounded file as long as output outpaces the cycle. Possible directions: fairness in the worker (re-offer the key instead of recursing), and/or not holding the thread lock across the flush, and/or a flush timeout in the close path.

Workaround

Reload/reopen the T3 window — the reconnect interrupts the hung close and releases the lock.

Start with packages/shared/src/KeyedCoalescingWorker.ts, then trace terminal.persistHistory, flushPersist, drainKey, and the terminal.close/open lock paths described in the report. Reproduce with a redraw-heavy terminal loop and inspect whether one busy key starves other work. Done means terminal.close completes promptly and a new terminal can open while another terminal emits output rapidly, without unbounded persistence cycling.