V2 session: auto-revert leaves forked session without instruction checkpoint

Summary

In OpenCode V2, a forked session can become unusable through supported UI and API operations after queued inputs overlap with tool execution and a revert occurs. Every later prompt is admitted and starts execution, but fails before a model response with SchemaError: Missing key at ["boundary"].

Environment

• opencode version: 0.0.0-next-16502
• OS: Linux x86_64, kernel 7.0.0-28-generic
• Terminal: Ghostty (TERM=xterm-256color, COLORTERM=truecolor)
• Shell: /usr/bin/zsh
• Install/channel: global npm installation, next
• Active plugins: intentionally excluded from this report

Reproduction

1. Create or continue a forked session.
2. While an assistant step is executing a tool, submit another user input so it is queued.
3. Continue execution until the queued input is promoted; submit another input while tool work is still in progress.
4. Let the provider request fail and retry. In the observed incident, the provider returned provider.internal with server_is_overloaded.
5. Interrupt the retried step so the session enters the revert flow.
6. Let the revert stage and commit.
7. Send any subsequent prompt in the session.

Expected Behavior

Queued inputs and the revert flow should preserve a valid instruction-state checkpoint. The session should continue normally, or present a recoverable error without corrupting future turns.

Actual Behavior

After the revert, every later prompt follows this pattern:

1. session.input.admitted
2. session.execution.started
3. session.execution.failed

No assistant text is produced. The server logs:

Failed to drain Session
SchemaError: Missing key
  at ["boundary"]
  ...
  at InstructionState.rebuild
  at InstructionState.observe
  at InstructionState.prepare
  at SessionRunner.callModel

Observed sequence around the first corruption:

2183 session.tool.input.started
2184 session.input.admitted
2185 session.tool.input.ended
2186 session.tool.called
2187 session.tool.success
2188 session.step.ended
2189 session.input.promoted
...
2208 session.input.admitted
...
2216 session.step.ended
2217 session.input.promoted
2219 session.retry.scheduled (provider.internal / server_is_overloaded)
2221 session.step.failed (aborted / Step interrupted)
2222 session.execution.interrupted
2223 session.revert.staged
2225 session.revert.committed
2226 session.input.admitted
2227 session.execution.started
2228 session.execution.failed (Missing key at ["boundary"])

The same input.admitted -> execution.started -> execution.failed sequence then repeated for every later prompt. The failure was deterministic and also prevented the built-in session fork operation.

Additional Context

At the time of diagnosis, the affected forked session had no row in instruction_state. Its 153 persisted messages remained readable, but rebuilding instruction state from its durable history failed at InstructionState.rebuild before a model request could start. The investigation did not capture the table state before the revert, so it is not known whether the revert removed a checkpoint or exposed a pre-existing missing checkpoint in the fork replay path.

The built-in session fork operation also failed with the same Missing key at ["boundary"] error, so normal recovery paths were unavailable.

The same deterministic Missing key at ["boundary"] failure was observed in another session. The incident detailed here occurred in a newly forked session running the current next build, so it was not limited to a session created by an older V2 version.

Unsupported recovery runbook for 0.0.0-next-16502

This is a record of the exact recovery used for this incident, not a supported public database migration. The internal schema may differ in any other build. Stop if any stated invariant differs. Direct SQLite repair can destroy session data; use a verified backup and do not run these steps while OpenCode is writing to the database.

The runbook intentionally does not inspect or print prompts, message contents, instruction contents, credentials, or provider secrets. Replace only the affected session ID below.

1. Isolate the database and create a consistent backup

Close every OpenCode client connected to the shared service, then set the incident variables:

export SESSION_ID='ses_REPLACE_WITH_AFFECTED_SESSION_ID'
export DB="${OPENCODE_DB:-$HOME/.local/share/opencode/opencode-next.db}"
export BACKUP="${DB%.db}.pre-session-repair-$(date -u +%Y%m%dT%H%M%SZ).db"

Stop the service before reading or writing with an external SQLite client:

opencode2 service stop

Do not run opencode2 api while repairing; it may start the service again. Verify the database exists, create a consistent backup, and validate both files:

test -f "$DB"
sqlite3 -readonly "$DB" 'PRAGMA integrity_check;'
sqlite3 "$DB" ".backup '$BACKUP'"
sqlite3 -readonly "$BACKUP" 'PRAGMA integrity_check;'

Both integrity checks must print exactly ok. Keep the backup until a real post-recovery turn succeeds.

2. Capture non-content baselines and derive the fork boundary

sqlite3 -readonly -header -column "$DB" "
SELECT
  (SELECT count(*) FROM session_message WHERE session_id='$SESSION_ID') AS messages,
  (SELECT count(*) FROM event WHERE aggregate_id='$SESSION_ID') AS events,
  (SELECT count(*) FROM session_pending WHERE session_id='$SESSION_ID') AS pending,
  (SELECT count(*) FROM instruction_state WHERE session_id='$SESSION_ID') AS checkpoints;
"

export PARENT_ID="$(sqlite3 -readonly "$DB" "
  SELECT json_extract(data, '$.parentID')
  FROM event
  WHERE aggregate_id='$SESSION_ID'
    AND seq=0
    AND type='session.forked.2';
")"

export PARENT_SEQ="$(sqlite3 -readonly "$DB" "
  SELECT json_extract(data, '$.parentSeq')
  FROM event
  WHERE aggregate_id='$SESSION_ID'
    AND seq=0
    AND type='session.forked.2';
")"

export EPOCH_START="$(sqlite3 -readonly "$DB" "
  SELECT max(seq)
  FROM event
  WHERE aggregate_id='$SESSION_ID'
    AND type='session.instructions.updated.2';
")"

export THROUGH_SEQ="$(sqlite3 -readonly "$DB" "
  SELECT max(seq)
  FROM event
  WHERE aggregate_id='$SESSION_ID';
")"

printf 'parent=%s parent_seq=%s epoch_start=%s through_seq=%s\n' \
  "$PARENT_ID" "$PARENT_SEQ" "$EPOCH_START" "$THROUGH_SEQ"

For this incident, the required invariants were:

exactly one session.forked.2 event at child sequence 0
checkpoints = 0
PARENT_ID is non-empty
PARENT_SEQ is an integer
EPOCH_START is an integer and EPOCH_START <= THROUGH_SEQ

Abort if any invariant differs. In particular, do not overwrite an existing checkpoint.

Pending inputs are significant. Starting the service may resume them and execute tools. If the pending count is nonzero and those inputs may cause external effects, leave the service stopped until their handling has been reviewed. Do not delete pending rows as part of this workaround.

3. Validate the replay algorithm against the parent's stored checkpoint

The state is a JSON object mapping instruction keys to 64-character content hashes. Reconstruct it by applying each session.instructions.updated.2 delta in ascending durable sequence with SQLite json_patch, which implements JSON Merge Patch semantics.

Run this read-only validation:

sqlite3 -readonly -header -column "$DB" "
WITH RECURSIVE
updates AS (
  SELECT
    row_number() OVER (ORDER BY seq) AS rn,
    json_extract(data, '$.delta') AS delta
  FROM event
  WHERE aggregate_id='$PARENT_ID'
    AND type='session.instructions.updated.2'
    AND seq <= (
      SELECT through_seq
      FROM instruction_state
      WHERE session_id='$PARENT_ID'
    )
),
replay(rn, state_json) AS (
  VALUES(0, json('{}'))
  UNION ALL
  SELECT replay.rn + 1, json_patch(replay.state_json, updates.delta)
  FROM replay
  JOIN updates ON updates.rn = replay.rn + 1
),
reconstructed AS (
  SELECT state_json
  FROM replay
  ORDER BY rn DESC
  LIMIT 1
),
stored AS (
  SELECT current_values
  FROM instruction_state
  WHERE session_id='$PARENT_ID'
)
SELECT
  (SELECT count(*) FROM updates) AS updates_replayed,
  NOT EXISTS (
    SELECT key, value FROM json_each((SELECT state_json FROM reconstructed))
    EXCEPT
    SELECT key, value FROM json_each((SELECT current_values FROM stored))
  )
  AND NOT EXISTS (
    SELECT key, value FROM json_each((SELECT current_values FROM stored))
    EXCEPT
    SELECT key, value FROM json_each((SELECT state_json FROM reconstructed))
  ) AS matches_parent_checkpoint;
"

matches_parent_checkpoint must be 1. In this incident, 24 parent updates replayed to exactly the seven keys and hashes stored by OpenCode. Abort if the parent has no checkpoint or the result is not 1; the assumptions behind this recovery do not hold.

4. Preview and validate the affected session's reconstructed state

Replay the parent only through the child's parentSeq, then replay the child's own instruction deltas. Parent events must be applied before child events regardless of their local sequence numbers.

sqlite3 -readonly -header -column "$DB" "
WITH RECURSIVE
source_events AS (
  SELECT 0 AS source_order, seq, json_extract(data, '$.delta') AS delta
  FROM event
  WHERE aggregate_id='$PARENT_ID'
    AND type='session.instructions.updated.2'
    AND seq <= $PARENT_SEQ
  UNION ALL
  SELECT 1, seq, json_extract(data, '$.delta')
  FROM event
  WHERE aggregate_id='$SESSION_ID'
    AND type='session.instructions.updated.2'
    AND seq <= $EPOCH_START
),
updates AS (
  SELECT
    row_number() OVER (ORDER BY source_order, seq) AS rn,
    delta
  FROM source_events
),
replay(rn, state_json) AS (
  VALUES(0, json('{}'))
  UNION ALL
  SELECT replay.rn + 1, json_patch(replay.state_json, updates.delta)
  FROM replay
  JOIN updates ON updates.rn = replay.rn + 1
),
reconstructed AS (
  SELECT state_json
  FROM replay
  ORDER BY rn DESC
  LIMIT 1
)
SELECT
  (SELECT count(*) FROM source_events) AS updates_replayed,
  (SELECT count(*) FROM reconstructed, json_each(reconstructed.state_json)) AS instruction_keys,
  (SELECT count(*)
   FROM reconstructed, json_each(reconstructed.state_json)
   WHERE json_each.type <> 'text' OR length(json_each.value) <> 64) AS invalid_hashes;
"

For this incident, the preview had exactly seven instruction_keys and zero invalid_hashes. Abort if those values differ; do not generalize this write to another state shape.

5. Insert the checkpoint in one guarded transaction

The service must still be stopped. EPOCH_START is the last instruction delta included in the reconstructed state. THROUGH_SEQ is the maximum durable event already present, including the revert and deterministic failed executions. Setting through_seq only to EPOCH_START is insufficient because OpenCode will replay the corrupt revert range and fail again.

sqlite3 "$DB" <<SQL
.bail on
BEGIN IMMEDIATE;

CREATE TEMP TABLE repair_guard (
  ok INTEGER NOT NULL CHECK (ok = 1)
);

INSERT INTO repair_guard
SELECT
  ((SELECT count(*) FROM instruction_state WHERE session_id='$SESSION_ID') = 0)
  AND ((SELECT max(seq) FROM event WHERE aggregate_id='$SESSION_ID') = $THROUGH_SEQ)
  AND ($EPOCH_START <= $THROUGH_SEQ);

WITH RECURSIVE
source_events AS (
  SELECT 0 AS source_order, seq, json_extract(data, '$.delta') AS delta
  FROM event
  WHERE aggregate_id='$PARENT_ID'
    AND type='session.instructions.updated.2'
    AND seq <= $PARENT_SEQ
  UNION ALL
  SELECT 1, seq, json_extract(data, '$.delta')
  FROM event
  WHERE aggregate_id='$SESSION_ID'
    AND type='session.instructions.updated.2'
    AND seq <= $EPOCH_START
),
updates AS (
  SELECT
    row_number() OVER (ORDER BY source_order, seq) AS rn,
    delta
  FROM source_events
),
replay(rn, state_json) AS (
  VALUES(0, json('{}'))
  UNION ALL
  SELECT replay.rn + 1, json_patch(replay.state_json, updates.delta)
  FROM replay
  JOIN updates ON updates.rn = replay.rn + 1
),
reconstructed AS (
  SELECT state_json
  FROM replay
  ORDER BY rn DESC
  LIMIT 1
)
INSERT INTO instruction_state (
  session_id,
  epoch_start,
  through_seq,
  initial_values,
  current_values
)
SELECT
  '$SESSION_ID',
  $EPOCH_START,
  $THROUGH_SEQ,
  reconstructed.state_json,
  reconstructed.state_json
FROM reconstructed
WHERE (SELECT count(*) FROM json_each(reconstructed.state_json)) = 7
  AND NOT EXISTS (
    SELECT 1
    FROM json_each(reconstructed.state_json)
    WHERE json_each.type <> 'text' OR length(json_each.value) <> 64
  );

INSERT INTO repair_guard VALUES (changes());

COMMIT;
SQL

The final guard makes the transaction fail unless exactly one checkpoint was inserted. Because EPOCH_START was the latest child instruction update and there were no later instruction deltas in this incident, initial_values and current_values were identical. Do not assume this for a session with later instruction updates.

6. Validate the database before restart

sqlite3 -readonly "$DB" 'PRAGMA integrity_check;'

sqlite3 -readonly -header -column "$DB" "
SELECT
  epoch_start,
  through_seq,
  (SELECT count(*) FROM json_each(initial_values)) AS initial_keys,
  (SELECT count(*) FROM json_each(current_values)) AS current_keys
FROM instruction_state
WHERE session_id='$SESSION_ID';

SELECT
  (SELECT count(*) FROM session_message WHERE session_id='$SESSION_ID') AS messages,
  (SELECT count(*) FROM event WHERE aggregate_id='$SESSION_ID') AS events,
  (SELECT count(*) FROM session_pending WHERE session_id='$SESSION_ID') AS pending;
"

The integrity check must be ok; epoch_start, through_seq, and both key counts must match the preview; message and event counts must not decrease.

7. Restart and perform an end-to-end verification

opencode2 service start
opencode2 service status
opencode2 api get /api/health

Open the repaired session and send one controlled, side-effect-free prompt. Verify all of the following:

the prompt produces an assistant response
a new session.execution.succeeded event is persisted
the log has no new SchemaError for the session
the database still passes PRAGMA integrity_check

In this incident, recovery preserved all 153 pre-repair messages. Subsequent validation reached session.execution.succeeded at durable sequence 2366, with 179 persisted messages and no pending inputs.

8. Roll back if validation fails

Stop the service before restoring. Preserve the failed repaired database for analysis, replace it with the consistent backup, and remove only sidecar files associated with the active database after the service has stopped:

opencode2 service stop
mv -- "$DB" "$DB.failed-repair-$(date -u +%Y%m%dT%H%M%SZ)"
rm -f -- "$DB-wal" "$DB-shm"
cp -- "$BACKUP" "$DB"
sqlite3 -readonly "$DB" 'PRAGMA integrity_check;'
opencode2 service start

Keep the backup until the restored service and sessions have been verified. Delete it only after an end-to-end successful turn.

This recovery bypasses the corrupt durable range rather than explaining why that range cannot be replayed. The underlying issue still appears to involve V2 fork replay, queued input promotion, and revert processing. A regression test covering a fork with queued input, provider retry, interruption, and revert may reproduce the invalid instruction-state boundary without relying on SQLite repair.