apache / apache/lucene

NGramTokenizer strips whitespace, with no option to keep leading and trailing whitespace [LUCENE-3979]

Open
#5,052 1 comment 0 reactions 0 assignees View on GitHub
affects-version:2.9.2 affects-version:3.0 legacy-jira-label:tokenizer legacy-jira-label:whitespace legacy-jira-priority:Minor module:analysis type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

org.apache.lucene.analysis.ngram.NGramTokenizer removes whitespace, making a search for literal strings like " test" and "test " equivalent to "test". Searching with relevant whitespace is sometimes desired, particularly where ngrams are used.

This could be fixed by either removing .trim() from the line shown below, or by providing a flag to specifically set trimming behaviour (keeping trim=true as the default so that existing code using this analyzer is not broken).

111: inStr = new String(chars).trim(); // remove any trailing empty strings

---
Migrated from [LUCENE-3979](https://issues.apache.org/jira/browse/LUCENE-3979) by David Mason
Environment:
```
n/a
```

Contributor guide

Open the contributing guide

Research direction

Start in org.apache.lucene.analysis.ngram.NGramTokenizer at the line 111 call to String(chars).trim(). Compare the requested preservation of leading and trailing whitespace with the stated need to keep trim=true by default for compatibility. Done means literal strings such as " test" and "test " are no longer treated as equivalent to "test", without breaking existing default behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.