[Bug]: Per-chunk assistant persistence causes progressive live output lag and rapid SQLite growth

Maintainer TL;DR: On T3 Code 0.0.31 with assistant streaming enabled, a 10,545-character response created 3,024 durable events (median delta: 3 characters). Across two Windows installations, clearing only T3 Code's internal history immediately restored responsiveness. On one installation, state.sqlite then regrew from 282 KB to 218 MB and 93,943 events in approximately 25 hours. PRAGMA quick_check remained ok. The proposed direction is to coalesce assistant deltas before persistence/publication, or use ephemeral live delivery with bounded periodic/final persistence.

Summary

With assistant streaming enabled, live Codex output becomes progressively slower as T3 Code accumulates conversation and tool-call history. The provider continues producing output normally, but the T3 Code UI falls increasingly far behind; even a short final response can take one or two minutes to finish appearing.

Two independent Windows installations show the same pattern. On both machines, removing only T3 Code's internal project/thread history and compacting SQLite restored responsive streaming immediately. On the second machine, the problem returned after only two threads: the database grew from 288 KiB to about 48 MiB and accumulated 11,526 thread.message-sent events for only 82 projected messages.

Read-only inspection confirms that individual assistant messages are split into thousands of durable events. One 10,545-character response produced 3,024 events whose median text delta was 3 characters. Concatenating those deltas exactly reproduced the projected message.

This appears to be live event-history amplification rather than provider latency, SQLite corruption, or ordinary free-page bloat.

Before submitting

• I searched existing issues and did not find a duplicate.
• I included enough detail to reproduce or investigate the problem.

Area

apps/server

Steps to reproduce

1. Run T3 Code Alpha 0.0.31 on Windows with the Codex provider.
2. Enable Settings → Assistant output (enableAssistantStreaming=true).
3. Start from a fresh or recently compacted %USERPROFILE%\.t3\userdata\state.sqlite.
4. Add a project and use two agentic threads for several normal turns, including tool calls and at least one moderately long assistant response.
5. Observe that provider output continues normally while text in T3 Code appears in tiny chunks with increasing delay.
6. Close T3 Code and inspect the database read-only:

PRAGMA quick_check;

SELECT event_type, COUNT(*)
FROM orchestration_events
GROUP BY event_type
ORDER BY COUNT(*) DESC;

SELECT COUNT(*) FROM orchestration_command_receipts;
SELECT COUNT(*) FROM projection_thread_messages;
SELECT COUNT(*) FROM projection_thread_activities;

For a deterministic high-amplification case, stream a roughly 10k-character assistant response from a provider that emits character/token-sized content.delta events. Compare the number of persisted thread.message-sent rows for its messageId with the final projected message length.

Expected behavior

Assistant output should remain responsive regardless of historical conversation volume. Provider deltas should be coalesced before durable persistence and client publication, or live deltas should use an ephemeral path with bounded periodic/final persistence.

The number of SQLite events, command receipts, projection writes, and UI publications should be bounded by a reasonable time/size cadence rather than being approximately proportional to the number of 2–4 character provider chunks.

Old, archived, or deleted threads and large tool payloads should not make new live output progressively slower.

Actual behavior

Each provider content.delta in streaming mode is dispatched as a thread.message.assistant.delta command. The decider converts it to a durable thread.message-sent event. The orchestration transaction then appends the event, updates projections, and upserts a command receipt. The live thread subscription publishes each event separately, and the client applies each live event as a one-item batch, so its adjacent-event coalescing does not combine them.

Provider runtime events first enter ProviderRuntimeIngestion's unbounded serial DrainableWorker. Each assistant delta waits for a complete orchestrationEngine.dispatch, so this is the primary queue where provider output can outpace persistence. OrchestrationEngine is also serialized and uses an unbounded command queue, making it the downstream persistence bottleneck rather than necessarily the only queue accumulating the backlog. When tiny provider deltas arrive faster than the event/receipt/projection/UI pipeline can complete, displayed text falls behind the provider.

Independent machine A: accumulated-history A/B test

Before cleanup:

Metric	Value
Main database size	335,400,960 bytes (~335 MB decimal)
T3 Code projects	12
Threads	63
Projected messages	1,431
Projected activities	20,814
Thread orchestration events	164,974
thread.message-sent events	141,922
thread.activity-appended events	20,991
Thread command receipts	164,682

PRAGMA quick_check returned ok.

A controlled cleanup was performed only after T3 Code and its background processes were stopped. A verified backup of state.sqlite, WAL/SHM files, and settings.json was created first. Only internal project/thread projections, runtime rows, project/thread events, and receipts were removed. Authentication, pairing, migrations, projection state, settings, provider configuration, project directories, Git repositories, source files, and worktrees were preserved.

After checkpoint and VACUUM:

Metric	Before	After
Main database size	335,400,960 B	282,624 B
Projects	12	0
Threads	63	0
Messages	1,431	0
Activities	20,814	0
Thread/project events	164,994	0
Thread/project receipts	164,702	0

Protected row counts and the settings.json hash remained unchanged, final quick_check was ok, and all existing project directories remained present. After T3 Code restarted and an existing project was re-added, streaming performance improved immediately and dramatically.

Independent machine B: rapid recurrence after cleanup

Before its cleanup, this installation had:

Metric	Value
Main database	158,072,832 B (150.75 MiB)
WAL	5,112,952 B
Projects	4
Threads	42
Projected messages	566
Projected activities	8,195
Orchestration events	57,102
thread.message-sent	47,714
thread.activity-appended	8,228
Command receipts	56,985

After the same protected cleanup/checkpoint/VACUUM, the database was 294,912 B (288 KiB), all target counts were zero, protected data was byte/count identical, and quick_check returned ok.

After only two threads and 11 turns, the lag returned:

Metric	Value
Main database	44,421,120 B (42.36 MiB)
WAL	6,002,872 B (5.73 MiB)
Combined DB/WAL/SHM	50,456,760 B (~48.12 MiB)
Threads	2
Turns	11
Projected messages	82
Projected activities	1,190
Orchestration events	12,800
thread.message-sent	11,526
thread.activity-appended	1,190
Command receipts	12,789

PRAGMA quick_check was still ok and freelist_count was 0, so this was active persisted data rather than corruption or unused-page bloat.

Payload-shape analysis without reading conversation content showed:

• 11,526 thread.message-sent events referenced only 82 distinct messageId values.
• 71 message IDs were repeated.
• The most amplified message had 3,024 events for 10,545 final characters.
• Its average delta was 3.49 characters; median 3; maximum 15.
• Other sampled messages had 1,139, 1,016, 488, 328, and 274 events.
• For every repeated projected message, concatenating persisted deltas exactly matched the final projected text.
• 361 tool.completed activities held about 11.36 million characters of content; the largest single payload was about 1.29 million characters, adding secondary persistence/projection pressure.

The same prompts through the provider CLI stream normally, so provider generation latency is not the primary cause.

Impact

Major degradation or frequent failure

Version or commit

T3 Code Alpha 0.0.31

Environment

Two independent Windows desktop installations; Codex provider; assistant streaming enabled; SQLite database at %USERPROFILE%\.t3\userdata\state.sqlite.

Logs or stack traces

Machine A, accumulated state:
  message-sent events:       141,922
  activity-appended events:   20,991
  thread command receipts:   164,682
  main DB:                335,400,960 B

Machine B, fresh-state recurrence after 2 threads / 11 turns:
  projected messages:             82
  message-sent events:         11,526
  command receipts:            12,789
  main DB + WAL + SHM:     50,456,760 B

One projected assistant message:
  final characters:             10,545
  persisted events:              3,024
  average delta characters:       3.49
  median delta characters:           3

SQLite quick_check before/after cleanup and after recurrence: ok
SQLite freelist_count after recurrence: 0

Screenshots, recordings, or supporting files

No attachments. Database inspection reported only aggregate counts, sizes, hashes, and payload lengths; no conversation content or secrets were extracted.

Workaround

Turning off Settings → Assistant output (enableAssistantStreaming=false) switches the server to buffered delivery and avoids per-chunk persistence for ordinary responses, at the cost of losing token-by-token output.

Deleting internal T3 Code project/thread history followed by a safe WAL checkpoint and VACUUM also restores performance temporarily, but the problem recurs and destructive manual SQLite maintenance should not be required.

Related reports and work

• #4008 — streaming assistant deltas rescan full thread activity history. Closely related server-side amplification, but the fresh-state recurrence above also demonstrates thousands of durable transactions for a single response.
• #4596 — large event backlogs cause quadratic replay when reopening a thread. Related consequence of the same durable event firehose; this report also covers live streaming before reopening/replay.
• #4349 — open PR to coalesce streaming Markdown renders.
• #4323 — closed, unmerged PR that coalesced assistant deltas before orchestration persistence (100 ms / 512 characters).

Suggested direction

• Coalesce high-frequency assistant deltas before persistence and broadcast using a bounded time/size cadence.
• Keep live ephemeral streaming separate from durable event history, persisting periodic checkpoints and the final message.
• Preserve exact final text and flush pending deltas at completion, approval/user-input boundaries, interruption, errors, and session exit.
• Bound/cap replay and use snapshot fallback for large resume gaps.
• Batch live and replay client state publications instead of publishing once per raw event.
• Avoid rescanning shell-summary/activity state for text-only deltas.
• Limit, externalize, or deduplicate very large tool-completion payloads.
• Add supported retention/compaction and a diagnostics view for DB size and event counts.

The key regression test should feed hundreds or thousands of one-character deltas and assert exact final-text preservation while keeping durable event count, command receipts, projection writes, and client publications bounded by the configured batching cadence.

Start in apps/server by tracing ProviderRuntimeIngestion into OrchestrationEngine and the live thread subscription; inspect how each assistant delta becomes a durable event and client publication. Reproduce with hundreds of tiny deltas and use the reported SQLite counts to measure amplification. Done means exact final text is preserved while durable events, receipts, projections, and publications remain bounded, with a regression test covering this behavior.