apache / apache/pinot

Text-index does not support multi-token substring search where first and last tokens are partial

Open
#10,863 5 comments 3 reactions 1 assignee Claimed by @chenboat View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
189

Description

We've seen many cases where a user wants to search for a substring in a field with a text index. If all of the tokens in the query are complete words, we can directly use a phrase search:
`SELECT * FROM table WHERE text_match("col", '"substring match query"')`
However, if the first or last token is a partial word (e.g., `"string match que"`), the query will not return any results. Treating the query as regex text-match query does not work either as Pinot only supports regex match on a single token.
To work around the limitation, we can use this a query like this:
`SELECT * FROM table WHERE text_match("col", '/*string/ AND match AND /que*/') AND "col" LIKE "%string match que%"`
But this is very slow and computationally expensive due to the LIKE which is necessary for validating the order of the tokens.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.