matrixorigin / matrixorigin/matrixone

[Performance]: safely push candidate LIMIT for exact FULLTEXT2 pure-MUST filters

Open
#27,251 1 comment 0 reactions 1 assignee Claimed by @Ariznawlll View on GitHub
kind/performance phase/testing severity/s0
Dominant language
Go
Stars
1.9k
Forks
311
Avg merge
1d 3h
Merged PRs (30d)
768

Description

## Background

FULLTEXT2 exact integer primary-key membership can safely bound the internal candidate stream for a literal pure-MUST Boolean query. Without the bound, a query with `LIMIT/OFFSET` may materialize many more candidates than the final page needs. Approximate Bloom membership and richer Boolean shapes cannot use this optimization because false positives or later Boolean evaluation can under-fill the page.

## Proposed scope

- Push `LIMIT + OFFSET` into the FULLTEXT2 table function only for one literal pure-MUST Boolean MATCH stream with an exact integer prefilter and pushdown enabled.
- Preserve the existing safe limit behavior when there is no residual filter.
- Keep VARCHAR/UUID/Bloom membership, prepared patterns, multiple MATCH streams, phrase, prefix, SHOULD, MUST-NOT, and other complex shapes unbounded at the table function.

## Correctness evidence

- BIGINT exact membership with `LIMIT 2 OFFSET 1` receives candidate limit 3 and returns the same page as pushdown OFF.
- A high-scoring filtered-out document does not cause under-fill.
- VARCHAR/Bloom has no table-function candidate limit and matches pushdown OFF.
- Complex Boolean, phrase, prefix, and MUST-NOT plans have no candidate limit.
- The dedicated FULLTEXT2 BVT passed 24/24 on an isolated candidate standalone.

## Non-goals

- No WAND, F1B, term statistics, filtered-scan reuse, INCLUDE fields, SQL syntax, index format, CDC, or runtime-filter wire-protocol changes.
- No claim of a search-kernel speedup; the observed benefit is reducing downstream candidates only.

## QA decision

- QA required: yes
- Reason: this is a planner/table-function correctness boundary involving LIMIT semantics and exact versus approximate membership.
- Production entrypoint: SQL `MATCH ... AGAINST (... IN BOOLEAN MODE)` with an exact integer WHERE prefilter and literal LIMIT/OFFSET.
- Automated terminal coverage: typed-plan tests, under-fill fixture, FULLTEXT2 table-function tests, and the dedicated distributed BVT.
- Remaining validation: exact-head CI and reviewer verification of the plan-shape boundary.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.