Add support for skipping/defaulting columns during managed offline flow rollup/dedup process
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 189
Description
Here is what I said in the pinot troubleshooting channel:
If this works as I think it might I could have the best of both worlds maybe. A certain amount of my data is in realtime table w/ the event_id for potential auditing, then as I move to offline table I default event_id to 0 and get good rollup.
To which @Jackie-Jiang repsonded:
It is absolutely reasonable. We don't support it currently, but it is doable. Essentially we need to add a new task config to skip some columns when running the task in ROLLUP or DEDUP mode. Internally we will fill default values to these columns so that they won't be considered.
Contributor guide
Research direction
Start by tracing the managed offline table flow and the task configuration used for ROLLUP and DEDUP modes. Identify where columns are selected and where default values are applied; done means a new configuration can skip selected columns during both modes while those columns receive default values and are excluded from rollup or deduplication.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100