clickhouse the data source support incremental checkpoints? or How can I implement resume broken transfers functionality
- Dominant language
- Java
- Stars
- 9.7k
- Forks
- 2.4k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 203
Description
I imported 100 million pieces of data and manually created checkpoints, but I have to wait for the task to complete and then create a full checkpoint。
(1) start a job
(2) During task execution, manually create checkpoints
./bin/seatunnel.sh -s {jobId}
(3) The checkpoint task waits for job to complete before it can succeed
dose clickhouse support incremental checkpoints?
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the checkpoint command entry point in bin/seatunnel.sh and investigate how ClickHouse jobs handle checkpoints during execution. Determine whether incremental checkpoints can be created before completion and used to resume broken transfers; the issue is done when that behavior and its supported scope are established.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- clickhouse, java
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100