Support time based segment split in flink connector
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 189
Description
## Goal
Allow consistent segment overwrite using the flink connector for time sorted data.
## Description
In order to support segment overwite for time-sorted data we need to implement segment split based on the time column ( the current implementation only takes the row count per segment split in account ).
### Reference
Follow up of https://apache-pinot.slack.com/archives/C01S5EHPS2U/p1655911946659589?thread_ts=1655654979.793919&cid=C01S5EHPS2U
Contributor guide
Research direction
No files or tests are named. Start by locating the Flink connector's current row-count-based segment-splitting implementation and review the linked Slack discussion for context. Done means time-column-based splitting supports consistent segment overwrite for time-sorted data.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100