apache / apache/pinot

Allow No Raw Data Index with S3 data store option for TEXT Index

Open
#6,317 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
189

Description

I would like to suggest a feature for TEXT_MATCH index

At the moment if we have "noRawDataForTextIndex": "true" , this will allow to store just index without data. Which is a great feature where we can reduce the size of the index. However the issue is that we wont be able to fetch the actual data itself but just do search on it. What I would like to have is something where we store the index without raw data also provide an option to store the raw data in a different store like S3. So when a search happens, it searched against the index and pulls the corresponding data from S3 and provide the results. This allow us to have a cheaper datastore plus efficient search.

Contributor guide

Open the contributing guide

Research direction

Start by tracing the TEXT_MATCH index and the noRawDataForTextIndex option in the Pinot codebase, then investigate how an S3-backed raw-data store could participate in search results. Done means defining and implementing the storage and retrieval behavior so searches use the index and return corresponding data from S3.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, java
Domain
databases, search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.