apache / apache/pinot

Add support for skipping/defaulting columns during managed offline flow rollup/dedup process

Open
#8,886 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
189

Description

Here is what I said in the pinot troubleshooting channel:
If this works as I think it might I could have the best of both worlds maybe. A certain amount of my data is in realtime table w/ the event_id for potential auditing, then as I move to offline table I default event_id to 0 and get good rollup.

To which @Jackie-Jiang repsonded:
It is absolutely reasonable. We don't support it currently, but it is doable. Essentially we need to add a new task config to skip some columns when running the task in ROLLUP or DEDUP mode. Internally we will fill default values to these columns so that they won't be considered.

Contributor guide

Open the contributing guide

Research direction

Start by tracing the managed offline table flow and the task configuration used for ROLLUP and DEDUP modes. Identify where columns are selected and where default values are applied; done means a new configuration can skip selected columns during both modes while those columns receive default values and are excluded from rollup or deduplication.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.