matrixorigin / matrixorigin/matrixone
[Performance]: reuse FULLTEXT2 term metadata within a query
- Dominant language
- Go
- Stars
- 1.9k
- Forks
- 311
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 768
Description
## Background
FULLTEXT2 query execution can decode the same loaded-segment term metadata more than once in a single query. Pure-MUST, BM25/WAND, legacy Boolean, prefix expansion, and exact live-DF calculation can independently call `LookupLoaded` for the same `(segment, term)` pair. The repeated FST/term-entry decode allocates metadata that is immediately discarded.
## Proposed scope
- Add a lazy query-local positive/negative cache in `globalStats` keyed by loaded segment and term.
- Route global live DF, pure-MUST, WAND, legacy term evaluation, and prefix expansion through the resolver.
- Keep build-side segments on direct lookup because their postings are already materialized.
- Keep cursor buffers and phrase execution state query-local and independently owned.
## Non-goals
- No cross-query cache, locks, eviction policy, session variable, index-format, DDL, FST, CDC, or runtime-filter protocol changes.
- No change to phrase-slot lookup or direct Segment APIs.
- No change to SQL semantics, query routing, result membership, or scores.
## Acceptance criteria
- Each loaded `(segment, term)` is decoded at most once per query, including negative lookups.
- PK membership and every PK's `float32` score bits remain unchanged for pure-MUST, BM25, TF-IDF, prefix, and complex Boolean paths.
- FULLTEXT2 package tests, focused/full race, build, vet, and relevant table-function tests pass.
- Microbenchmarks demonstrate lower metadata lookup allocations and latency across 1/2/4/8 terms and 1/4 segments.
- A same-data Boolean C10 comparison is collected after the stacked P0/P0.1 base is merged and the branch is rebased to current main.
## QA decision
- QA required: yes
- Reason: this is an internal semantics-preserving query-kernel optimization; the performance claim requires exact-head CI and a deployed workload check.
- Production entrypoint: positional FULLTEXT2 `MATCH ... AGAINST (... IN BOOLEAN MODE)` and related Boolean modes.
- Automated terminal coverage: query-local cache tests, differential score/membership tests, package race tests, and table-function tests.
- Remaining validation: rebase after issue #27245 / PR #27246 merges, then collect the planned paired mo-55 C10 evidence.
Contributor guide
Assessment
This issue has not been assessed yet.