[Bug]: Startup OOM with oversized replay batches remains unaddressed by #10777

Before submitting

• I searched existing issues and did not find a duplicate.
• I included enough detail to reproduce or investigate the problem.

Area

apps/server

Steps to reproduce

1. Launch nightly 1387 with my existing profile. The GUI opens and existing threads are usable.
2. Update to nightly 1400 and launch against the same profile. The backend repeatedly crashes before a usable GUI appears. I originally encountered this after clicking the in-app update button.
3. Restore 1387 without deleting history or manually editing projection cursors. The same profile becomes usable again.

This is profile-dependent. The relevant condition is historical session events containing unusually large validation-error strings. A focused local test reproduced heap exhaustion when reading a single oversized replay page, without retaining earlier pages. Filtering for deletion/revert events in SQL before loading full payloads completed. This was a focused replay test, not a full patched-desktop test.

Expected behavior

Updating T3 Code should preserve access to my existing projects and threads. Startup attachment cleanup should process the events it needs with bounded memory, even when historical session records contain oversized errors. An old input-validation error should not make a later release unable to launch.

Actual behavior

I cannot run the affected recent T3 Code nightlies with my existing history. The GUI never becomes usable because the backend repeatedly exhausts its JavaScript heap and aborts. I have been staying on 1387 to keep working. Sequential install tests confirmed that 1377 and 1387 work, 1400 fails, and rolling back to 1387 works again. Builds 1486 and 1507 also failed during the investigation.

Follow-up to #10774 and #10777

This is a follow-up to #10774 and its duplicate #10872, focusing on an oversized single-batch failure that the merged fix #10777 does not address.

#10777 fixes retention of previously consumed event pages by replacing recursive concatenation with Stream.paginate. However, it still reads and decodes up to 500 full event records per page. It does not bound page size in bytes or filter out events unrelated to attachment cleanup before loading their payloads. Git ancestry checks confirm that failing builds 1486 and 1507 already include #10777.

In a focused local replay test, reading a single oversized page alone caused heap exhaustion. There were no earlier pages to retain. Filtering for deletion/revert events in SQL before loading full payloads completed. This distinguishes the failure reproduced here from the consumed-page retention fixed by #10777.

Original regression and remaining cause

The startup regression began between 1387 and 1400 with #9871, commit 7220dfe2c949476eaa7d21eccbcd3a0ce0eddb49 (feat(chat): attach files to question answers).

In apps/server/src/orchestration/Layers/ProjectionPipeline.ts, the new attachment-cleanup bootstrap starts from zero for a missing cleanup cursor and calls eventStore.readFromSequence(cleanupStart, Number.MAX_SAFE_INTEGER). It filters for deletion/revert events only after full event payloads have been fetched and decoded.

The reader in apps/server/src/persistence/Layers/OrchestrationEventStore.ts loads 500 full rows per page. A row-count limit does not bound memory when individual payloads are very large. An OOM prevents the cleanup cursor from advancing, so the next launch repeats the failure.

Historical oversized errors

The affected history contains validation errors that embedded the rejected input itself. Those errors were persisted as session.lastError and carried into subsequent session-history records.

This is a historical data problem: testing the default schema formatter shipped with Effect 4.0.0-rc.112 confirmed that it no longer embeds the rejected input by default. I am not claiming the current formatter still produces these giant messages. The existing records remain and still need to be handled safely during replay. Bounding newly persisted error summaries would be a separate preventive measure.

What has and has not been verified

The release rollback, source comparison, and focused single-page reproduction support this diagnosis. The replay test used the Effect Stream implementation extracted from 1400 with JSON.parse in place of application schema validation. I have not tested #10777 alone in a patched desktop build against my profile, captured a full-app heap profile, or verified a production fix. The conclusion about its remaining gap is based on the actual PR diff and the independent single-page failure.

Suggested follow-up

A dedicated cleanup query could filter event types in SQL and read only the necessary deletion/revert metadata, while preserving cursor advancement and retry semantics. General replay should also account for oversized batches rather than relying only on a row-count limit. Preventing new oversized records alone would not restore startup for affected existing histories.

Impact

Blocks work completely

Version or commit

Last working: 0.0.41-nightly.20260908.1387. First failing: 0.0.41-nightly.20260908.1400. Also failed: 0.0.41-nightly.20260910.1486 and 0.0.41-nightly.20260910.1507. Introducing change: #9871 / 7220dfe2.

Environment

macOS, Apple Silicon / arm64. Official T3 Code Nightly desktop builds downloaded from GitHub; existing local profile.

Logs or stack traces

# Official desktop build: 0.0.41-nightly.20260908.1400
# Two consecutive backend startup attempts.
# Sanitized excerpt: timestamps, process IDs and addresses omitted.
# These are 1400 logs, not a test of #10777 in isolation.

# Attempt 1
5445 ms: Mark-Compact (reduce) 3888.2 (3891.7) -> 3886.6 (3891.7) MB, pooled: 0.0 MB, 19.48 / 0.00 ms (average mu = 0.881, current mu = 0.003) last resort; GC in old space requested
5466 ms: Mark-Compact (reduce) 3886.6 (3891.7) -> 3886.5 (3889.5) MB, pooled: 0.0 MB, 21.39 / 0.00 ms (average mu = 0.770, current mu = 0.000) last resort; GC in old space requested
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory

# Attempt 2
5901 ms: Mark-Compact (reduce) 3888.2 (3894.0) -> 3886.7 (3894.0) MB, pooled: 0.0 MB, 19.71 / 0.00 ms (average mu = 0.889, current mu = 0.002) last resort; GC in old space requested
5923 ms: Mark-Compact (reduce) 3886.7 (3894.0) -> 3886.6 (3892.0) MB, pooled: 0.0 MB, 21.80 / 0.00 ms (average mu = 0.786, current mu = 0.000) last resort; GC in old space requested
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory

Screenshots, recordings, or supporting files

No response

Workaround

Rolling back to the official nightly 1387 restores the GUI and access to my existing threads without deleting history or manually advancing projection cursors. Nightly 1377 and stable 0.0.40 also worked for me. Updating to the affected builds makes the app unusable again.

Diagnosis used Codex assistance, source comparison, read-only inspection, and focused replay tests. No production fix was applied during this investigation.

Start in apps/server/src/orchestration/Layers/ProjectionPipeline.ts and trace the attachment-cleanup bootstrap through apps/server/src/persistence/Layers/OrchestrationEventStore.ts and readFromSequence. Reproduce the focused oversized-page failure, then verify that startup cleanup handles the affected historical records without exhausting the JavaScript heap and can advance its cursor.