Colliding migration IDs silently skip migrations; one undecodable event row bricks startup

What happened

Running either t3 start or t3 serve failed immediately with:

ERROR (#5): PersistenceDecodeError: Decode error in OrchestrationEventStore.readFromSequence:decodeRows: Composite(Pointer(Composite(Pointer(AnyOf()))))

The desktop app crash-looped its backend child every ~40 seconds, exit code 1 each time, and never came up.

The trigger was mine: I built an AppImage from a community fork branch to try Copilot CLI support, not realising that branch was based on the unreleased orchestration-v2 work. But the fork only exposed the problem. Once that build had touched the database, stable v0.0.37 was permanently unbootable and could not repair itself, and the two mechanisms that let that happen are both in this repo and both fail silently.

Diagnosis

Both failures were reproduced from scratch on a clean, fully migrated v0.0.37 install (details under Steps to reproduce), so neither depends on the fork.

1. readFromSequence decodes rows it never filters, and one bad row is terminal

apps/server/src/persistence/Layers/OrchestrationEventStore.ts:185-188 reads the event log with no discriminator beyond the sequence cursor:

FROM orchestration_events
WHERE sequence > ${request.sequenceExclusive}
ORDER BY sequence ASC
LIMIT ${request.limit}

The whole page is then decoded as a batch against OrchestrationEventPersistedRowSchema, whose type is OrchestrationEventType. Any row whose event_type is outside that union fails the entire batch.

Because ProjectionPipeline.ts:1778-1779 starts each projector from its stored lastAppliedSequence, and that watermark only advances when a batch decodes successfully, the same poisoned batch is re-read on every boot forever. There is no skip, no quarantine, and no way out without hand-editing SQLite.

In my case the projectors were parked at 4626 and the very next page was:

index	sequence	event_type	in the v1 union?
0	4627	thread.created	yes
1	4628	message.updated	no

which is exactly the reported at [1]["type"].

2. Migrator matches applied migrations by ID and never by name

Migrator.run takes only the highest applied ID and skips everything at or below it (effect-smol packages/effect/src/unstable/sql/Migrator.ts:250):

if (currentId <= latestMigrationId) {
  continue
}

Names are recorded in effect_sql_migrations but never compared. So any build that occupies your ID range marks your own migrations as done. My ledger held:

ID	name in my DB	name in v0.0.37
41	OrchestrationV2	AuthSessionClientConnection
42	OrchestrationV2Subagents	ProjectionThreadLinkedPullRequest
43	OrchestrationV2Foundation	ProjectionThreadsUnsettledAt
44-49	OrchestrationV2*, ApplicationEventSource, ScheduledTasks, LegacyV1ImportState	(do not exist)

IDs 1-40 matched v0.0.37 byte for byte, so the divergence began exactly where the two lines forked. v0.0.37's own 041/042/043 therefore never ran, and these columns were simply absent:

• auth_sessions.client_surface, auth_sessions.client_app_version
• projection_threads.linked_pull_request_json, projection_threads.unsettled_at

Nothing reported this. The Migrations ran successfully line is only logged when migrations actually execute, so a fully skipped set produces no log output at all. Once the decode crash was cleared, the very first thread-list query died on it:

PersistenceSqlError: SQL error in ProjectionSnapshotQuery.getCommandReadModel:listThreads:query
  [cause]: Error: no such column: linked_pull_request_json

So this is a second, independent brick behind the first.

Why the two combine badly

ApplicationEventSource adds orchestration_events.application_event_version plus the index idx_orchestration_events_application_sequence on (application_event_version, sequence) — an index that exists precisely so readers can filter by that column. The v1 reader never learned to use it, so the two event generations share one table with nothing separating them.

Worth noting for anyone assuming version numbers are a safety net: the build that did this self-reported as 0.0.33, i.e. older than the 0.0.37 I moved back to. A version string says nothing about which migration line a database is on.

Steps to reproduce

Neither repro needs the fork. Both were run against a clean v0.0.37 install created in a scratch HOME, and both fail on every subsequent boot, not just the first.

A. One unknown event type bricks the server (reproduces the reported error exactly)

On an otherwise empty, fully migrated database, insert a single row:

INSERT INTO orchestration_events
 (event_id, aggregate_kind, stream_id, stream_version, event_type, occurred_at,
  command_id, causation_event_id, correlation_id, actor_kind, payload_json, metadata_json)
VALUES
 ('repro-unknown-type','thread','00000000-0000-4000-8000-000000000001',0,
  'message.updated','2026-08-31T00:00:00.000Z',NULL,NULL,NULL,'server','{}','{}');

Then t3 serve. Result: exit 1 with the identical readFromSequence:decodeRows error, and no server is ready. Repeated twice — same outcome, since the watermark cannot advance. One row out of one is enough; so is one row out of 4,624.

B. Three phantom ledger rows silently disable three migrations

Rewind a clean database to the pre-041 state and let a foreign line claim those IDs:

DELETE FROM effect_sql_migrations WHERE migration_id >= 41;
ALTER TABLE projection_threads DROP COLUMN unsettled_at;
ALTER TABLE projection_threads DROP COLUMN linked_pull_request_json;
ALTER TABLE auth_sessions DROP COLUMN client_surface;
ALTER TABLE auth_sessions DROP COLUMN client_app_version;
INSERT INTO effect_sql_migrations (migration_id, created_at, name) VALUES
 (41,'2026-08-28 13:45:59','OrchestrationV2'),
 (42,'2026-08-28 13:45:59','OrchestrationV2Subagents'),
 (43,'2026-08-28 13:45:59','OrchestrationV2Foundation');

Then t3 serve. Result: exit 1 on no such column: linked_pull_request_json, with no migration-related log line whatsoever. All four columns are still missing afterwards. The migrator considers the schema current.

How I actually got there (context, not required to reproduce)

1. Built T3-Code-0.0.33-x86_64.AppImage from copilot-v2 on a public fork, to try Copilot CLI support. That branch carries migrations 041_OrchestrationV2 … 049_LegacyV1ImportState — the exact nine names, at the exact nine IDs, now in my ledger. Its PR against this repo has since been closed as not ready.
2. Ran it once, 2026-08-28 13:45:59. It applied 35-40 (matching main) then 41-49 (its own), and LegacyV1ImportState appended 402 rows to orchestration_events tagged application_event_version = 2, using v2-only types: message.updated ×197, turn-item.updated ×197, thread.metadata-updated ×2, thread.visited ×2, plus 4 rows whose type names happen to be valid in v1.
3. Went back to stable v0.0.37 — permanently unbootable from then on.

On the same orchestration-v2 branch today these migrations are renumbered 044-052, i.e. rebased past main's 041-043. The collision was a snapshot-in-time hazard, which is exactly why detection matters rather than discipline.

Suggested fix

1. Filter the read. Give readFromSequence (and readAll) an application_event_version predicate; idx_orchestration_events_application_sequence already exists for it.
2. Make an undecodable row survivable. Skip or quarantine it with a loud warning instead of failing the batch. As #8789 argues, a single malformed event in an append-only log should never make the service unbootable — and because the watermark is gated on batch success, today it is unbootable permanently.
3. Verify migration identity, not just height. Compare recorded names against loaded ones for every applied ID and refuse to start on a mismatch, naming the offending IDs. A clear "this database was migrated by a different build" beats a no such column crash six steps downstream. If upstream Migrator will not do it, a cheap preflight against migrationManifest in Migrations.ts would.
4. Log the no-op case. Migrations ran successfully only appears when something ran; silence currently means both "nothing to do" and "everything was skipped".

Happy to split this into separate issues for the reader and the migrator if you would rather track them apart — I filed together because one database ends up wedged by both at once, and either alone would have been recoverable.

Version

0.0.37 (crash). Database previously migrated by a self-built AppImage labelled 0.0.33 from a fork's copilot-v2 branch.

Environment

Linux x64, kernel 7.0.0-30-generic, Node 24.20.0, sqlite3 3.46.1. Desktop AppImage against a local server, plus t3 serve from a terminal; both fail identically. No systemd service. Providers: claude, copilot.

Evidence

# Reported failure, every boot (paths and home directory redacted)
[08:25:01.241] ERROR (#5): PersistenceDecodeError: Decode error in
  OrchestrationEventStore.readFromSequence:decodeRows: Composite(Pointer(Composite(Pointer(AnyOf()))))
  [cause]: SchemaError: Expected "project.created" | "project.meta-updated" | ... | "thread.activity-appended"
    at [1]["type"]

# Desktop backend crash-loop, ~40s apart, all exit 1
12:55:32 backend child process failure output start  pid=... port=3773
12:55:32 backend child process output  stdout  [07:55:03.439] ERROR (#5): PersistenceDecodeError: ...
12:55:32 backend child process failure output end    code=1
... repeats through 13:04:52 ...

# Event types present, by application_event_version. v=2 rows were unreadable by v0.0.37.
v  event_type                            n     min_seq  max_seq
1  thread.activity-appended              3285  19       4623
1  thread.message-sent                   840   6        4618
1  thread.session-set                    468   8        4624
1  (9 further v1 types)                  ...
2  project.created                       2     4625     4626     <- valid name, from the import
2  thread.created                        2     4627     4633     <- valid name, from the import
2  message.updated                       197   4628     5023     <- rejected
2  turn-item.updated                     197   4629     5024     <- rejected
2  thread.metadata-updated               2     4632     4638     <- rejected (v1 has thread.meta-updated)
2  thread.visited                        2     5025     5026     <- rejected

# All nine projectors parked immediately before the first rejected row
projector                         last_applied_sequence
projection.projects               4626
projection.threads                4626
(... 7 more, all 4626)

# Ledger: IDs 1-40 match v0.0.37 exactly; 41-49 are a different migration line
41  2026-08-28 13:45:59  OrchestrationV2                          <- v0.0.37 expects AuthSessionClientConnection
42  2026-08-28 13:45:59  OrchestrationV2Subagents                 <- v0.0.37 expects ProjectionThreadLinkedPullRequest
43  2026-08-28 13:45:59  OrchestrationV2Foundation                <- v0.0.37 expects ProjectionThreadsUnsettledAt
44  2026-08-28 13:45:59  OrchestrationV2ProviderSessionBindings
45  2026-08-28 13:45:59  OrchestrationV2ThreadLaunchWorkflows
46  2026-08-28 13:45:59  ApplicationEventSource
47  2026-08-28 13:45:59  OrchestrationV2EffectCancellation
48  2026-08-28 13:45:59  ScheduledTasks
49  2026-08-28 13:45:59  LegacyV1ImportState

# Second brick, surfaced only after the decode crash was cleared
ERROR (#5): PersistenceSqlError: SQL error in
  ProjectionSnapshotQuery.getCommandReadModel:listThreads:query
  [cause]: Error: no such column: linked_pull_request_json

# Proof the three migrations really had not run
projection_threads.unsettled_at            : absent
projection_threads.linked_pull_request_json: absent
auth_sessions.client_surface               : absent
auth_sessions.client_app_version           : absent

Related issues

#8789 — same function and same "startup decoder rejects a persisted row" shape, but triggered by an out-of-enum origin.surface value rather than an out-of-union event_type, and it does not involve the migrator. Not a duplicate; its suggested fix 3 would have prevented this crash, which is why I would treat both as one class. #4518 is the same class again for a settings value. #4374 covers nightly-to-stable downgrade trouble but reports hidden chats, not a crash. #7537 (closed) also concerns readFromSequence limits, but the capping behaviour, not decoding.

Fix applied or workaround

Rolled the foreign migration line back out of the database, after an integrity-checked sqlite3 .backup snapshot, in one transaction:

DELETE FROM orchestration_events WHERE application_event_version = 2;       -- 402 rows
DELETE FROM orchestration_command_receipts WHERE command_type <> 'legacy';  -- 2 rows
UPDATE projection_state SET last_applied_sequence = 4624 WHERE last_applied_sequence > 4624;
DROP INDEX IF EXISTS idx_orchestration_events_application_sequence;
ALTER TABLE orchestration_events DROP COLUMN application_event_version;
ALTER TABLE orchestration_command_receipts DROP COLUMN command_type;
DROP TABLE ...;                          -- 24 orchestration_v2_* tables, plus scheduled_tasks
DELETE FROM effect_sql_migrations WHERE migration_id >= 41;
UPDATE sqlite_sequence SET seq = 4624 WHERE name = 'orchestration_events';

Reading the nine branch migrations first was necessary rather than optional: ApplicationEventSource also adds command_type to orchestration_command_receipts, an existing v1 table, which is easy to miss. Reassuringly it only ever INSERTs into orchestration_events — never UPDATE or DELETE — so no original event row was modified and the v1 log came back intact.

v0.0.37 then applied 41_AuthSessionClientConnection, 42_ProjectionThreadLinkedPullRequest, 43_ProjectionThreadsUnsettledAt and started cleanly; verified over two consecutive cold boots with zero decode errors. All 4,624 v1 events, 2 projects, 2 threads and 197 messages survived.

This needed a schema-level diff of the two migration lines to get right. A user without that would reasonably conclude the profile was lost.

Filed by

Claude Fable 5 (claude-fable-5) via t3 triage, running in Claude Code.

Start with apps/server/src/persistence/Layers/OrchestrationEventStore.ts, ProjectionPipeline.ts, effect-smol Migrator.run, and migrationManifest in Migrations.ts. Run the supplied SQL reproductions and start the server; done means both failure paths are detected or survivable, with diagnostics covering event-version handling and migration identity mismatches.