matrixorigin / matrixorigin/matrixone

[Performance]: reuse FULLTEXT2 term metadata within a query

Open
#27,250 2 comments 0 reactions 1 assignee Claimed by @VioletQwQ-0 View on GitHub
kind/performance severity/s0
Dominant language
Go
Stars
1.9k
Forks
311
Avg merge
1d 3h
Merged PRs (30d)
768

Description

## Background

FULLTEXT2 query execution can decode the same loaded-segment term metadata more than once in a single query. Pure-MUST, BM25/WAND, legacy Boolean, prefix expansion, and exact live-DF calculation can independently call `LookupLoaded` for the same `(segment, term)` pair. The repeated FST/term-entry decode allocates metadata that is immediately discarded.

## Proposed scope

- Add a lazy query-local positive/negative cache in `globalStats` keyed by loaded segment and term.
- Route global live DF, pure-MUST, WAND, legacy term evaluation, and prefix expansion through the resolver.
- Keep build-side segments on direct lookup because their postings are already materialized.
- Keep cursor buffers and phrase execution state query-local and independently owned.

## Non-goals

- No cross-query cache, locks, eviction policy, session variable, index-format, DDL, FST, CDC, or runtime-filter protocol changes.
- No change to phrase-slot lookup or direct Segment APIs.
- No change to SQL semantics, query routing, result membership, or scores.

## Acceptance criteria

- Each loaded `(segment, term)` is decoded at most once per query, including negative lookups.
- PK membership and every PK's `float32` score bits remain unchanged for pure-MUST, BM25, TF-IDF, prefix, and complex Boolean paths.
- FULLTEXT2 package tests, focused/full race, build, vet, and relevant table-function tests pass.
- Microbenchmarks demonstrate lower metadata lookup allocations and latency across 1/2/4/8 terms and 1/4 segments.
- A same-data Boolean C10 comparison is collected after the stacked P0/P0.1 base is merged and the branch is rebased to current main.

## QA decision

- QA required: yes
- Reason: this is an internal semantics-preserving query-kernel optimization; the performance claim requires exact-head CI and a deployed workload check.
- Production entrypoint: positional FULLTEXT2 `MATCH ... AGAINST (... IN BOOLEAN MODE)` and related Boolean modes.
- Automated terminal coverage: query-local cache tests, differential score/membership tests, package race tests, and table-function tests.
- Remaining validation: rebase after issue #27245 / PR #27246 merges, then collect the planned paired mo-55 C10 evidence.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.