fix(runtime-host): cold-start artifact recovery sweeps O(all files) realpath/lstat, blocking Host readiness for minutes
- Dominant language
- TypeScript
- Stars
- 5.4k
- Forks
- 502
- Avg merge
- 1d 2h
- Merged PRs (30d)
- 715
Description
### What happened
Cold-starting a Runtime Host against a workspace with a large accumulated artifact store blocks Host readiness for ~10 minutes at 60-70% CPU. Connected TUI clients render nothing during this window (`maka --resume ` appears to hang).
Observed on a long-lived workspace with ~11.7k artifact files (~6 GB on disk, comparable `artifact_records` row count). A long-running Host amortizes these costs in memory; when the Host is replaced (for example a version upgrade with a compatibility-epoch cutover), the new Host pays the full cost synchronously during startup recovery.
CPU profile of the Host main thread during the stall (CDP `Profiler`, 500µs sampling):
- `preparePurgePathsUnlocked` (`packages/storage/dist/artifact-store.js:528`)
- `resolveArtifactRemovalEntry` → `realpath` / `lstat` per record (`node:internal/fs/promises`)
- `decodeArtifactRecordJsons` (`artifact-metadata-codec.js`) — full in-memory decode of all ~11.7k records
- Followed by `readEventsForRecovery` / `readSqliteAgentRunEvents` replaying 56k `core_agent_run_events` + 29k `session_messages` before serving
Root causes in `packages/storage/src/artifact-store.ts`:
1. Startup recovery (`hasCanonicalRecoveryResidueUnlocked`, `recoverMetadataTempsUnlocked`, publication recovery) `readdir`s every session directory and runs `realpath` + `lstat` on **every artifact file** — O(files) syscalls on every Host start.
2. `preparePurgePathsUnlocked` performs its referential-integrity check by resolving `resolveArtifactRemovalEntry` (realpath) for **all records**, not just the purge set — O(all records) per `purge()` call, so retiring M sessions costs M × N realpaths.
3. The artifact store eagerly decodes all metadata records into memory at open (`metadataRepository.readAll()`).
Severity scales with artifact count. Fresh/small installs are unaffected (sub-millisecond sweeps), but heavy long-lived workspaces degrade on every Host restart (version update, epoch cutover, crash recovery), and the TUI gives no progress indication while it waits.
### How to reproduce
1. Accumulate a workspace artifact store with ~10k files (long-lived heavy use: many sessions, side conversations, tool outputs).
2. Stop the running Runtime Host (or trigger an epoch cutover via a version upgrade).
3. Run `maka --resume `.
4. Observe: new Host spins at 60-70% CPU for ~10 minutes; TUI stays blank until Host recovery completes.
Profiling one-liner used:
```sh
kill -USR1 # enable inspector on 127.0.0.1:9229
# then CDP Profiler.start/stop over ws://127.0.0.1:9229/
```
### Environment
- Commit: `4cc781f31` (main, 2026-08-27)
- Node: 26.3.0
- OS: Linux
- Surface: Runtime Host / TUI
### Logs, screenshots, or additional context
Top self-time frames from the profile (5s window during the stall):
```
12.1% run
5.2% run
1.3% realpath
1.1% lstat
1.0% lstat node:internal/fs/promises:1670
0.9% realpath node:internal/fs/promises:1826
0.8% preparePurgePathsUnlocked packages/storage/dist/artifact-store.js:528
0.6% decodeArtifactRecordJsons packages/storage/dist/artifact-metadata-codec.js:40
```
(libuv threadpool fs syscalls account for additional CPU not visible to the main-thread inspector.)
Suggested directions:
- Make startup recovery lazy or index-driven (metadata in SQLite) instead of per-file `realpath`/`lstat` sweeps.
- In `preparePurgePathsUnlocked`, restrict referential-integrity checks to records sharing the purge set's relative paths (metadata query) rather than resolving every record.
- Surface Host bootstrap progress to waiting TUI clients so a cold start does not look like a hang.
Contributor guide
Research direction
Start in packages/storage/src/artifact-store.ts, tracing hasCanonicalRecoveryResidueUnlocked, recoverMetadataTempsUnlocked, publication recovery, preparePurgePathsUnlocked, and metadataRepository.readAll(); compare the profile with packages/storage/dist/artifact-store.js. Reproduce with a large artifact store and the cold-start/resume flow, then inspect the SQLite metadata and event paths. Done means startup and purge no longer perform O(all files/records) realpath/lstat work, and Host readiness/TUI resume avoids the reported multi-minute stall.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- node.js, sqlite, typescript
- Domain
- backend, cli, databases, performance
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100