Support checkpointing in Pinot Flink Connector
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 2d 55m
- Merged PRs (30d)
- 182
Description
Current the PinotSinkFunction implements the CheckpointedFunction interface, but just throws an exception during checkpointing. This make it impossible to use the pinot connector in any Flink job that requires checkpointing. I believe checkpointing could be supported in the PinotSinkFunction but storing all segment records in Flink state until the segment is committed to Pinot.
Contributor guide
Research direction
Start by reading PinotSinkFunction and its CheckpointedFunction implementation to understand the current checkpoint exception and how Flink state would relate to segment commits in Pinot. Define completion as allowing checkpointed Flink jobs to use the connector while retaining segment records in state until the segment is committed to Pinot; the issue does not name tests or additional files.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100