expression: NGRAM wildcard terms match nothing unless exactly ngram_token_size long
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Bug Report
An NGRAM wildcard term matches nothing unless it happens to be exactly `ngram_token_size` characters long. Raised in review of #70484.
### Reproduction
With `ngram_token_size = 2`, against a document containing `abcd`:
| query | `MatchesNothing()` | matches `abcd` | expected |
| --- | --- | --- | --- |
| `+abc*` | true | false | should match |
| `+a*` | true | false | should match, as a prefix search |
| `+ab*` | false | true | correct |
| `+abc` (no wildcard) | false | true | correct |
Only a wildcard term of exactly the gram size behaves. Longer and shorter both collapse the whole query to match-nothing, because a required clause that normalizes to nil makes the enclosing query unsatisfiable.
### Cause
`normalizePrefixTerm` in `pkg/expression/fulltext/query.go` grams the term and then requires exactly one gram:
```go
tokens := ngramFilter(sourceTokens, parserInfo.ngramTokenSize, parserInfo.ngramTokenSize)
tokens = lowerFilter(tokens)
if len(tokens) != 1 {
return nil
}
return prefixNode{prefix: tokens[0].Text}
```
`abc*` grams to `ab`, `bc` - two tokens, so nil. `a*` is shorter than the gram size and grams to nothing - also nil.
The non-wildcard path does not have this problem: it builds a phrase from multiple grams, which is why `+abc` works while `+abc*` does not.
### Proposed change
Per review: build a phrase node when the term produces several grams, and a prefix node for the short or single-gram cases.
**Confirm against MySQL before implementing.** What MySQL does for a wildcard term shorter than the gram size, and whether the multi-gram case is an ngram phrase or something looser, should come from MySQL's behaviour or documentation rather than from this issue text - the distinction changes which rows match.
### Tests
`pkg/expression/fulltext/query_test.go` currently asserts that `a*` and `abc*` do not match, pinning the present behaviour. Those expectations need updating as part of the fix, and are worth reading first: they document the bug rather than the intent.
Contributor guide
Research direction
Start with normalizePrefixTerm in pkg/expression/fulltext/query.go and the related assertions in pkg/expression/fulltext/query_test.go. Confirm MySQL’s behavior for wildcard terms shorter than the ngram size and for multiple grams before choosing the expected matching semantics. Update the tests so a*, abc*, and exact-size terms reflect the confirmed behavior and pass the fulltext query test suite.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, mysql
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100