apache / apache/seatunnel

clickhouse the data source support incremental checkpoints? or How can I implement resume broken transfers functionality

Open
#10,344 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
9.7k
Forks
2.4k
Avg merge
3d 13h
Merged PRs (30d)
203

Description

I imported 100 million pieces of data and manually created checkpoints, but I have to wait for the task to complete and then create a full checkpoint。

(1) start a job

(2) During task execution, manually create checkpoints

./bin/seatunnel.sh -s {jobId}

(3) The checkpoint task waits for job to complete before it can succeed

dose clickhouse support incremental checkpoints?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the checkpoint command entry point in bin/seatunnel.sh and investigate how ClickHouse jobs handle checkpoints during execution. Determine whether incremental checkpoints can be created before completion and used to resume broken transfers; the issue is done when that behavior and its supported scope are established.

Written by the indexing model from the issue text.

Assessment

Tech stack
clickhouse, java
Domain
data-engineering, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.