Allow No Raw Data Index with S3 data store option for TEXT Index
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 189
Description
I would like to suggest a feature for TEXT_MATCH index
At the moment if we have "noRawDataForTextIndex": "true" , this will allow to store just index without data. Which is a great feature where we can reduce the size of the index. However the issue is that we wont be able to fetch the actual data itself but just do search on it. What I would like to have is something where we store the index without raw data also provide an option to store the raw data in a different store like S3. So when a search happens, it searched against the index and pulls the corresponding data from S3 and provide the results. This allow us to have a cheaper datastore plus efficient search.
Contributor guide
Research direction
Start by tracing the TEXT_MATCH index and the noRawDataForTextIndex option in the Pinot codebase, then investigate how an S3-backed raw-data store could participate in search results. Done means defining and implementing the storage and retrieval behavior so searches use the index and return corresponding data from S3.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, java
- Domain
- databases, search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100