A search that scans is a bug: put mail, recall and chat on the index
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 434
- Forks
- 21
- Avg merge
- 18m
- Merged PRs (30d)
- 326
Description
The rule
Nothing that answers a query may iterate a store. Not a service, not the record, not a page. A full scan is correct until it is slow, and by the time it is slow it is load-bearing in six places and the fix is a migration rather than a change.
This is not a new capability. internal/data/sqlite.go is a real FTS5 index with a LIKE fallback and relevance scoring, and half the services already use it. The half that do not are the two largest.
Where it stands
| how it searches | |
|---|---|
social, news, archive, chat (rooms) |
FTS5 via data.Search ✅ |
service/mail |
for _, msg := range messages — scans every message on the instance |
internal/thread (behind service/recall) |
strings.Contains over every message of every thread |
service/images |
userdb.List with a where clause — a query, at least |
service/chat store |
scans an account's history (bounded at 2000, but the same shape) |
Mail's is the worst. Search iterates the package-level messages slice — every account's mail — and filters by owner as it goes. That is the same O(whole store) shape as #1464, where delivery measured 71ms over 5,000 messages and got worse than linearly. Search has the same curve and nobody has measured it.
Recall's matters most. It is what an agent calls to remember, so it runs on a model's initiative rather than a person's, and it runs often.
Why FTS5 before embeddings
Vectors are the more interesting answer and the wrong first move.
- Pure vector search is bad at what a mailbox is searched for: a name, an order number, a Message-ID, "the DMARC report from Google". Every system that shipped vector-only added keyword search back. What works for recall is hybrid — FTS and vector merged — which means FTS is needed either way.
- Embeddings cost a model call per message. Indexing a mail store means embedding every arriving email: real money, and a latency spike on the delivery path.
- FTS5 is already here. Wired for four services, no new dependency, nothing per message, and it takes these from O(n) to an index.
So: index first, measure whether recall is actually failing on semantics, and only then add vectors — as a second index beside the same database (sqlite-vec), not instead of the first.
Scope
service/mail— index on write, search the index. Also fixes searching another account's mail out of the scan by construction rather than by a filter inside the loop.internal/thread— same, which is whatservice/recallreads.service/chat— the store added in b5445caa. Small today and the same shape.- A test that a
Searchmethod does not range over a package-level store, so the next one is caught when it is written rather than when it is slow. - A line in AGENTS.md, since this is a rule and not a task.
Not in scope
Embeddings, until 1–3 land and there is a measurement saying keyword search is what is failing.
Done when
Searching a mailbox with 50,000 messages in it costs the same as searching one with 500, and a test fails if a new service reintroduces a scan.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with internal/data/sqlite.go and the existing data.Search callers, then trace search paths in service/mail, internal/thread, service/chat, and service/recall. Review the relevant store and search tests, add coverage that rejects package-level store scans, and update AGENTS.md. Done means these searches use the index and mailbox search scales similarly for 500 and 50,000 messages.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, sqlite
- Domain
- backend, databases, search
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100