pingcap / pingcap/tidb

expression: NGRAM wildcard terms match nothing unless exactly ngram_token_size long

Open
#70,533 0 comments 0 reactions 0 assignees View on GitHub
component/expression severity/moderate sig/planner type/bug
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Bug Report

An NGRAM wildcard term matches nothing unless it happens to be exactly `ngram_token_size` characters long. Raised in review of #70484.

### Reproduction

With `ngram_token_size = 2`, against a document containing `abcd`:

| query | `MatchesNothing()` | matches `abcd` | expected |
| --- | --- | --- | --- |
| `+abc*` | true | false | should match |
| `+a*` | true | false | should match, as a prefix search |
| `+ab*` | false | true | correct |
| `+abc` (no wildcard) | false | true | correct |

Only a wildcard term of exactly the gram size behaves. Longer and shorter both collapse the whole query to match-nothing, because a required clause that normalizes to nil makes the enclosing query unsatisfiable.

### Cause

`normalizePrefixTerm` in `pkg/expression/fulltext/query.go` grams the term and then requires exactly one gram:

```go
tokens := ngramFilter(sourceTokens, parserInfo.ngramTokenSize, parserInfo.ngramTokenSize)
tokens = lowerFilter(tokens)
if len(tokens) != 1 {
return nil
}
return prefixNode{prefix: tokens[0].Text}
```

`abc*` grams to `ab`, `bc` - two tokens, so nil. `a*` is shorter than the gram size and grams to nothing - also nil.

The non-wildcard path does not have this problem: it builds a phrase from multiple grams, which is why `+abc` works while `+abc*` does not.

### Proposed change

Per review: build a phrase node when the term produces several grams, and a prefix node for the short or single-gram cases.

**Confirm against MySQL before implementing.** What MySQL does for a wildcard term shorter than the gram size, and whether the multi-gram case is an ngram phrase or something looser, should come from MySQL's behaviour or documentation rather than from this issue text - the distinction changes which rows match.

### Tests

`pkg/expression/fulltext/query_test.go` currently asserts that `a*` and `abc*` do not match, pinning the present behaviour. Those expectations need updating as part of the fix, and are worth reading first: they document the bug rather than the intent.

Contributor guide

Open the contributing guide

Research direction

Start with normalizePrefixTerm in pkg/expression/fulltext/query.go and the related assertions in pkg/expression/fulltext/query_test.go. Confirm MySQL’s behavior for wildcard terms shorter than the ngram size and for multiple grams before choosing the expected matching semantics. Update the tests so a*, abc*, and exact-size terms reflect the confirmed behavior and pass the fulltext query test suite.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, mysql
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.