matrixorigin / matrixorigin/matrixone

[Performance]: defer FULLTEXT2 phrase position decoding until document alignment

Open
#27,155 2 comments 0 reactions 1 assignee Claimed by @Ariznawlll View on GitHub
phase/testing severity/s0
Dominant language
Go
Stars
1.9k
Forks
311
Avg merge
1d 3h
Merged PRs (30d)
768

Description

## Background

FULLTEXT2 positional phrase search currently decodes a posting block's position payload while cursors are still advancing and aligning document ordinals. For sparse phrase intersections, many decoded position blocks are never inspected because the terms do not align on a common document.

Profiles from the 2,303,643-document gojieba workload show `phraseCursor.skipTo`, position decoding, and varint decoding as the dominant Natural Language Mode costs.

## Proposed scope

- Split phrase-cursor document-block state from position-block state.
- Let `doc()` and `skipTo()` decode only document ordinals.
- Decode positions only after all phrase cursors align on a common document and `positions()` is requested.
- Decode each position block at most once per cursor/block.
- Preserve the existing phrase membership, BM25/TF-IDF score, result ordering contract, buffer ownership, and serialized index format.

## Non-goals

- No SQL, DDL, FST, posting, CDC, or runtime-filter format changes.
- No changes to Boolean, BM25 OR, prefix, or position-free routing.
- No new cache or background state.

## Acceptance criteria

- Differential phrase tests match the existing fallback for build-side and serialized loaded segments, including sparse intersections and block boundaries.
- Document-only cursor advancement does not decode positions.
- FULLTEXT2 package tests, focused race tests, build, vet, and the relevant FULLTEXT2 BVT control pass on the PR head.
- A same-data A/B shows lower CPU/request or higher Natural Language Mode QPS without a Boolean control regression.

## QA decision

- QA required: yes
- Reason: the optimization is internal and semantics-preserving, but the performance claim depends on a deployed real-data workload.
- Production entrypoint: positional FULLTEXT2 `MATCH ... AGAINST (... IN NATURAL LANGUAGE MODE)`.
- Automated terminal coverage: differential PK membership and float32 score-bit tests.
- Remaining validation: representative latest-main A/B on the 2,303,643-document workload.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.