apache / apache/maka

fix(runtime-host): Host recovery decodes every Session's full history before readiness, breaching the 45s election deadline

Open
#4,032 6 comments 0 reactions 0 assignees View on GitHub
bug help wanted
Dominant language
TypeScript
Stars
5.4k
Forks
502
Avg merge
1d 2h
Merged PRs (30d)
716

Description

### What happened

Runtime Host startup recovery decodes the entire durable execution history of every recoverable Session before reporting readiness. On a long-lived workspace (~80 recoverable Sessions, ~500 runs, ~100k event/message rows in `runtime.sqlite`) recovery takes ~150s of mostly main-thread CPU, which breaches the client's default 45000ms election deadline: `maka --resume ` fails with "did not become ready before the startup deadline elapsed" (`lastRegistration.state: "recovering"`, `readyWaitFailed: 1`) even though the Host is making progress. Retrying after the Host settles works.

This is the residual bottleneck after the artifact-store costs in #4027 are removed; CPU profile of a recovering Host (CDP Profiler on the main thread) shows:

- ~32% `readSqliteAgentRunEvents` (`packages/storage/dist/agent-run-store.js`)
- ~30% better-sqlite3 `all()` calls
- `decodeStoredRuntimeEvent`, GC pressure

Root causes in `packages/runtime-host/src/server/hosted-execution-recovery.ts` (`prepareHostedExecutionRecovery`):

1. **Discarded per-run validation reads.** For every Session, for every run, it awaits `readEventsForRecovery` + `readRuntimeEvents` and discards the results — a full decode of every event of every run, purely as a fail-closed consistency check.
2. **Messages are decoded twice.** `listForRecovery()` (`packages/storage/src/session-store.ts`) already calls `readMessagesForRecovery` per header and discards the result; `prepareHostedExecutionRecovery` then re-reads the same messages per Session.
3. The per-session loop is fully serial, though note better-sqlite3 `all()` and JSON decoding are synchronous main-thread work, so concurrency alone would not help — the fix has to reduce the amount of work, not interleave it.

Because admission/message repair decisions only involve Sessions with admissions, non-terminal runs, or pending closures, decoding the history of long-settled Sessions appears unnecessary for correctness — it is startup-time insurance with O(total history) cost paid on every Host start (upgrade, epoch cutover, crash recovery).

### How to reproduce

1. Use a workspace with a large accumulated history (many recoverable Sessions and runs; here ~80 Sessions / ~500 runs / ~100k rows).
2. Restart the Runtime Host (upgrade or stop the previous one).
3. Run `maka --resume ` within the first ~2 minutes.
4. Observe the election deadline error while the Host is still recovering; Host main thread sits at ~50-70% CPU decoding SQLite rows.

### Environment

- Commit: `d27c02af1` (main)
- Node: 26.3.0
- OS: Linux
- Surface: Runtime Host / TUI

### Logs, screenshots, or additional context

Election diagnostic from the client:

```json
{"deadlineMs":45000,"elapsedMs":45008,"candidateLaunches":1,"sawEndpointConnected":true,
"observations":{"readyWaitFailed":1,"connected":1},
"lastRegistration":{"state":"recovering","lifecycleMode":"ephemeral"}}
```

Related: #4027 (artifact-store cold-start costs; fix in #4031).

Suggested directions (needs a design decision on the startup-validation contract):

- Scope recovery to Sessions that actually need it (admissions, non-terminal runs, pending closures) instead of every recoverable Session; or make full-history validation lazy/on-demand.
- Drop the discarded double read of Session messages (keep a single decode).
- Optionally surface recovery progress so waiting clients can extend their deadline adaptively instead of failing at 45s.

Contributor guide

Open the contributing guide

Research direction

Start in packages/runtime-host/src/server/hosted-execution-recovery.ts at prepareHostedExecutionRecovery, then trace listForRecovery and readMessagesForRecovery in packages/storage/src/session-store.ts. Reproduce recovery with a large runtime.sqlite history and inspect the discarded validation and duplicate message reads. Done means the startup-validation contract is defined and recovery avoids unnecessary full-history decoding while preserving admission, non-terminal-run, and pending-closure repair decisions.

Written by the indexing model from the issue text.

Assessment

Tech stack
sqlite, typescript
Domain
backend, databases, performance
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.