matrixorigin / matrixorigin/matrixone

[Performance]: avoid full posting materialization for FULLTEXT2 pure-MUST queries

Open
#27,245 1 comment 0 reactions 1 assignee Claimed by @Ariznawlll View on GitHub
kind/performance phase/testing
Dominant language
Go
Stars
1.9k
Forks
311
Avg merge
1d 3h
Merged PRs (30d)
768

Description

## Background

FULLTEXT2 pure-MUST Boolean queries currently fall back to the full Boolean evaluator. That path allocates dense per-document score and MUST-hit state. In addition, exact global live document frequency is computed by materializing every loaded posting list before checking segment liveness.

Profiles from the 2,303,643-document gojieba workload show the dense evaluator and posting materialization dominating CPU and allocation cost under concurrent `boolean_must` queries.

## Proposed scope

- Route Boolean queries containing only single-term MUST clauses to an ordered posting-cursor intersection.
- Select the smallest segment posting list as the driver and advance other cursors with `skipTo`.
- Share membership cursors for duplicate terms while preserving each clause's original score contribution order.
- Count exact live term DF without materializing the complete posting list.
- Preserve liveness, WHERE membership, BM25/TF-IDF scoring, LIMIT behavior, and complex Boolean fallback.

## Non-goals

- No SQL, DDL, index-format, FST, CDC, planner, or runtime-filter protocol changes.
- No routing changes for phrase, prefix, group, SHOULD, MUST-NOT, ADJUST, or no-LIMIT queries.
- No cross-query cache, WAND change, or filter-driver implementation.

## Acceptance criteria

- Differential tests preserve PK membership and every PK's float32 score bits for build-side and serialized loaded segments.
- Duplicate and missing terms, BM25/TF-IDF, liveness, base plus CDC tail, delete/superseded copies, INCLUDE, and WHERE membership are covered.
- Complex Boolean shapes continue to use the existing evaluator.
- Loaded dirty-segment live DF uses O(BlockSize) temporary memory; fully live segments do not decode postings.
- FULLTEXT2 package tests, focused/full race, build, vet, and relevant table-function tests pass on the PR head.
- A same-data 2,303,643-document A/B demonstrates lower CPU/allocation cost or higher Boolean MUST QPS without a Natural Language Mode control regression.

## QA decision

- QA required: yes
- Reason: this is an internal, semantics-preserving search-kernel optimization, but its performance claim depends on a deployed real-data workload.
- Production entrypoint: positional FULLTEXT2 `MATCH ... AGAINST ('+term1 +term2 ...' IN BOOLEAN MODE)`.
- Automated terminal coverage: differential PK membership and float32 score-bit tests plus loaded/build and liveness variants.
- Remaining validation: exact-head CI and independent review of the production performance evidence.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.