apache / apache/pinot

Support checkpointing in Pinot Flink Connector

Open
#9,997 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 55m
Merged PRs (30d)
182

Description

Current the PinotSinkFunction implements the CheckpointedFunction interface, but just throws an exception during checkpointing. This make it impossible to use the pinot connector in any Flink job that requires checkpointing. I believe checkpointing could be supported in the PinotSinkFunction but storing all segment records in Flink state until the segment is committed to Pinot.

Contributor guide

Open the contributing guide

Research direction

Start by reading PinotSinkFunction and its CheckpointedFunction implementation to understand the current checkpoint exception and how Flink state would relate to segment commits in Pinot. Define completion as allowing checkpointed Flink jobs to use the connector while retaining segment records in state until the segment is committed to Pinot; the issue does not name tests or additional files.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.