text_match operator fails to execute query containing stop words
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 2d 55m
- Merged PRs (30d)
- 182
Description
Lucene strips away stop words and symbols prior to indexing but it seems like Pinot doesn't do the same when running queries on a text index. As a result, a query like:
`SELECT * FROM table WHERE text_match("col", '"function not in list"')`
will not return any result if the words `not` and `in` are stop words that were stripped out during ingestion. A temporary workaround is to exclude all stop words in the index config.
Contributor guide
Research direction
Start by reproducing the text_match query shown in the issue against an index that strips stop words during ingestion, then trace the text_match operator and its Lucene query handling. Done means the query still returns the matching row when words such as “not” and “in” were removed from the text index, without requiring the workaround of excluding stop words from the index configuration.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases, search
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100