perf(storage): batch multi-Session artifact purge during session retirement (O(M×N) guard scans)
- Dominant language
- TypeScript
- Stars
- 5.4k
- Forks
- 502
- Avg merge
- 1d 2h
- Merged PRs (30d)
- 715
Description
## Problem
Runtime Host session retirement drains a batch of M Sessions (`packages/runtime-host/src/server/session-retirement-coordinator.ts`), and each Session's artifact cleanup triggers its own fully serialized `ArtifactStore.purge()`. Each individual purge:
1. Runs a guard scan over **all** live records (case-insensitive path + inode/symlink alias validation),
2. Resolves removal entries (`realpath`/`lstat`) for every non-target live record,
3. Commits a full metadata rewrite (currently `DELETE` + re-INSERT of every record — see the companion metadata proposal (#4037)).
So retiring M Sessions against a store of N records costs M full-record scans and M full-table metadata commits — O(M×N) on both the filesystem and the metadata side. On a real store with ~11.7k records and hundreds of retiring Sessions, this dominates cold start (profile data in [this #4027 comment](https://github.com/apache/maka/issues/4027#issuecomment-5442149594): ~90 s of the residual ~130 s).
#4031 made per-purge resolution concurrent (8-wide worker pool), which is a constant-factor improvement only — the review there identified batched purge as the converged design.
## Proposed direction
A bulk purge path that accepts the whole retiring Session batch at once, reusing the existing multi-ID purge intent mechanism:
- **One purge intent** covering the union of the batch's artifact ids (the intent schema already carries multiple ids),
- **One guard scan** over the live record set (case/alias validation against the union of targets),
- **One unlink plan**, with bounded-concurrency resolution inside the batch (the #4031 worker pool),
- **One metadata commit** per batch instead of M.
This removes the M multiplier without weakening case-insensitive or symlink-alias integrity and without adding another authority. Combined with change-tracked metadata write-back (companion proposal), a retirement batch becomes one O(N) guard scan + one O(changed) metadata commit.
## Questions for maintainers
1. The retirement coordinator currently drains Sessions one at a time; is batching the artifact purge across the whole drain batch acceptable from the lifecycle/ownership point of view, or are there per-Session ordering guarantees the single-purge loop is protecting?
2. The purge intent record is the crash-recovery evidence for interrupted purges. Does carrying a multi-Session batch in one intent change any recovery-contract expectations (e.g. partial-batch resume semantics), or is "resume the whole intent" already the defined behavior?
3. Sequencing: this composes with the metadata change-tracking proposal (#4037) but does not depend on it. Prefer landing them separately (purge batching first removes the M metadata commits for the retirement path), or as one design?
Related: #4027 (cold-start investigation), #4031 (bounded purge resolution; review point on the O(M×N) structure and this converged design).
_This issue was prepared with AI assistance (Kimi k3-256k), including profiling and analysis._
Contributor guide
Research direction
Start with packages/runtime-host/src/server/session-retirement-coordinator.ts and the existing ArtifactStore.purge() path. Read the multi-ID purge intent mechanism, the bounded-concurrency work from #4031, and the metadata proposal in #4037 before resolving the lifecycle and recovery questions. Done means the retirement batch has defined ordering and recovery semantics and avoids repeated full scans and metadata commits.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- backend, performance
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100