apache / apache/pinot

text_match operator fails to execute query containing stop words

Open
#10,865 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 55m
Merged PRs (30d)
182

Description

Lucene strips away stop words and symbols prior to indexing but it seems like Pinot doesn't do the same when running queries on a text index. As a result, a query like:
`SELECT * FROM table WHERE text_match("col", '"function not in list"')`
will not return any result if the words `not` and `in` are stop words that were stripped out during ingestion. A temporary workaround is to exclude all stop words in the index config.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the text_match query shown in the issue against an index that strips stop words during ingestion, then trace the text_match operator and its Lucene query handling. Done means the query still returns the matching row when words such as “not” and “in” were removed from the text index, without requiring the workaround of excluding stop words from the index configuration.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases, search
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.