matrixorigin / matrixorigin/matrixone
[Bug]: FULLTEXT2 INCLUDE CHAR predicates ignore trailing-space comparison semantics
- Dominant language
- Go
- Stars
- 1.9k
- Forks
- 311
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 768
Description
## Description
FULLTEXT2 pushes predicates on an included `CHAR` column into `fulltext2_search` and compares the stored values byte-for-byte. SQL `CHAR` equality ignores trailing spaces, so the pushed predicate changes the row set: it produces false negatives for `=` / single-value `IN` and false positives for `<>`.
The same wrong row set is returned both by the covered TVF-only plan and by a plan that projects a non-covered column and joins back to the base table. The predicate has already discarded rows inside `fulltext2_search`; the later join cannot restore them.
## Environment
- Branch: `main`
- Commit: `01d60e1c4ded1b0f3fc1a4ecd75ce54e95e23b90`
- Local deployment: 1 Log / 1 TN / 2 CN; direct CN endpoints
- Date: 2026-09-17
## Steps to reproduce
```sql
SET experimental_fulltext2_index = 1;
CREATE DATABASE ft2_char_padding;
USE ft2_char_padding;
CREATE TABLE t(
id BIGINT PRIMARY KEY,
body TEXT,
ch CHAR(4),
note TEXT
);
INSERT INTO t VALUES
(1, 'alpha document', 'a', 'n1'),
(2, 'alpha document', 'a ', 'n2'),
(3, 'alpha document', 'a ', 'n3'),
(4, 'alpha document', 'abc', 'n4'),
(5, 'alpha document', 'abc ','n5'),
(6, 'alpha document', '', 'n6');
CREATE FULLTEXT2 INDEX ft ON t(body) INCLUDE(ch);
ALTER TABLE t ALTER REINDEX ft FULLTEXT2 FORCE_SYNC;
-- Base-table CHAR semantics: 1,2,3.
SELECT id FROM t
WHERE body LIKE '%alpha%' AND ch = 'a'
ORDER BY id;
-- FULLTEXT2 INCLUDE pushdown: only 1.
SELECT id FROM t
WHERE MATCH(body) AGAINST('+alpha' IN BOOLEAN MODE)
AND ch = 'a'
ORDER BY id;
-- Base table excludes 1,2,3; FULLTEXT2 incorrectly returns 2 and 3 as well.
SELECT id FROM t
WHERE MATCH(body) AGAINST('+alpha' IN BOOLEAN MODE)
AND ch <> 'a'
ORDER BY id;
-- Projecting a non-covered column still returns only 1 for ch='a'.
SELECT id, ch, note FROM t
WHERE MATCH(body) AGAINST('+alpha' IN BOOLEAN MODE)
AND ch = 'a'
ORDER BY id;
```
## Actual behavior
For `ch = 'a'`:
- base-table predicate: `1,2,3`;
- FULLTEXT2 query: `1`.
Changing the literal to `'a '` returns only row `2`, and changing it to `'a '` returns only row `3`, although all three literals compare equal to all three rows under the base table's `CHAR` semantics.
For `ch <> 'a'`, the base table returns `4,5,6`, while FULLTEXT2 returns `2,3,4,5,6`.
`EXPLAIN` for the covered query contains only `fulltext2_search` below the sorts. Adding `note` creates a base-table join but keeps the same wrong row set because the `CHAR` predicate remains pushed into the search operator.
## Expected behavior
Pushing a predicate into FULLTEXT2 must preserve the base table's typed comparison semantics. `CHAR` values that differ only by trailing spaces must compare equal for `=`, `<>`, and `IN` exactly as they do outside the index path.
## Controls and scope
- `RTRIM(ch) = 'a'` stays residual on the base table and returns `1,2,3` through the same FULLTEXT2 index.
- A prepared statement with dynamic `ch = ?` also returns `1,2,3`, because the dynamic predicate is not serialized as the literal in-index predicate.
- Equivalent predicates over included signed/unsigned integer extremes, NULLs, and `VARCHAR` values containing quotes or backslashes match the base-table result.
- Reproduced in 3 independent databases on both CN endpoints: 6/6 identical reproductions.
- The issue is deterministic and independent of data volume; it does not require big-data, stability, or chaos coverage.
## Code analysis
The planner admits both `VARCHAR` and `CHAR` literals for FULLTEXT2 include pushdown in `pkg/sql/plan/filter_predicate.go`. The FULLTEXT2 evaluator then implements string equality and ordering with `bytes.Equal` / `bytes.Compare` in `pkg/fulltext2/include_predicate.go`.
That comparison is valid for binary `VARCHAR` values but not for SQL `CHAR` trailing-space semantics. The safe fix is either to preserve typed `CHAR` comparison in the index evaluator or to keep `CHAR` predicates residual until the index representation carries the information needed to reproduce SQL comparison semantics.
Contributor guide
Research direction
Start with pkg/sql/plan/filter_predicate.go and pkg/fulltext2/include_predicate.go, then run the SQL reproduction queries for CHAR equality, inequality, and IN against base-table and FULLTEXT2 plans. Done means pushed predicates preserve SQL CHAR trailing-space semantics, including covered and joined plans, without changing the documented control cases.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, sql
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100