hoangsonww / hoangsonww/Diffuse-Native-Apps

Rebuild SearchIndex incrementally instead of scanning every snapshot on every query

Open
#5 0 comments 0 reactions 1 assignee Claimed by @hoangsonww View on GitHub
accessibility bug documentation enhancement good first issue help wanted question
Dominant language
Swift
Stars
2
Forks
0
Avg merge
1h 53m
Merged PRs (30d)
14

Description

## Motivation

`SearchIndex` (`Packages/DiffuseCore/Sources/DiffuseCore/Search/SearchIndex.swift`) materialises one `Entry` per snapshot, per section, and per entity into a flat array, then `search(_:limit:)` `compactMap`s across the whole array scoring every entry on each keystroke. Index construction requires every snapshot to be decoded and held in memory at once.

With the shipped defaults — 90 days of retention at a four-hour cadence — a Mac library reaches roughly 540 snapshots, each with a dozen capabilities and, for the applications and processes collectors, hundreds of entities. That is comfortably six figures of entries rebuilt from scratch and rescored per query. On watchOS and on older iPhones this is both a latency and a memory-pressure problem, and it scales linearly with how long someone has been using the product.

## Proposed Solution

Persist the index rather than deriving it per session. Add an on-disk inverted index next to `index.json` in the snapshot store, keyed by lowercased term to posting lists of `SearchTarget`. `SnapshotService` updates it incrementally on save and on retention delete, so the cost is per-snapshot rather than per-library. Keep the existing scoring but evaluate it only against candidate postings for the query terms rather than every entry.

Treat the index as a rebuildable cache — same contract as `index.json` — so a corrupt or absent file triggers a background rebuild instead of an error.

## Acceptance Criteria

- [ ] A persisted inverted index lives in the store and is updated on save and on retention delete
- [ ] `search` consults posting lists rather than scanning all entries; results are identical to today's for the existing test corpus
- [ ] Deleting the index file causes a transparent rebuild with no user-visible error
- [ ] A benchmark test covers a synthetic 500-snapshot library and asserts a bounded query time
- [ ] Peak memory during search no longer scales with library size
- [ ] The Kotlin `SearchIndex` gets the equivalent treatment or an explicit issue is filed for it

## Additional Context

`Documentation/Storage.md` describes `index.json` as rebuildable; the search index should carry the same guarantee. Skill: `search`.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.