apache / apache/doris-flink-connector

[Enhancement] How can I set the startup_mode to synchronize from the specified timestamp?

Open
#468 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
385
Forks
279
Avg merge
2d 2h
Merged PRs (30d)
6

Description

### Search before asking

- [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues.

### Description

I have a very large table and for some reason the synchronous redo log is overwritten, so i missing the kafka offset; the situation is that I can manually fix the data myself and I don t want to sync the historical data;
So how can I start syncing the data from the specified time point?

### Solution

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the existing startup_mode behavior and how Kafka offsets are handled in the Doris Flink connector. Clarify how a specified timestamp should map to the synchronization start point, then verify that synchronization begins there without replaying earlier historical data.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kafka
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.