apache / apache/pinot

Text Index lag time for realtime consuming segments

Open
#6,431 21 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
189

Description

I am having Text Index and and inverted Index . And for some data Text Index not able to pull out results. for the same search Inverted Index works all the time.

Eg: - The Text Index Search for following is not working
select * from mylog where text_match(log, '1DB7BAD2-406A-D773-B1DA-291664314C4F')

but it works for
select * from mylog where text_match(log, '0D82F520-62C8-9914-14B8-4C2331E54075')

it looks like somehow some data are skipped from text_index. However both search will works fine with regexp_like inverted index.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the two SQL queries against the same realtime-consuming segments, comparing text_match results with regexp_like on the example UUIDs. Investigate the Text Index and inverted-index paths for skipped segment data; done means text_match returns the same expected records consistently for both examples.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, sql
Domain
databases, search
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.