alibaba / alibaba/DataX

DataX支持增量同步吗?我想改造Reader.Task的startRead方法完成增量同步好像会出现尾部的一部分数据无法导入

Open
#1,401 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
17.4k
Forks
5.7k
PR merge metrics
No merged PRs in 30d

Description

想自己改一下源码支持增量同步 改造源码的如下,就是通过在reader内通过一个死循环,并且使用递增的id作为条件查询最新的数据,但是目前有个问题就是最新的一批数据会出现无法导入目标数据库的情况 个人猜测是因为batchSize没有达到导致的,请问有什么办法解决这个问题吗,就是在reader的这个循环内调用一下类似于IO流中flush() 这样的方法手动让这部分导入目标数据库
![GGVrFqA8BX](https://user-images.githubusercontent.com/37692792/173783154-7cc564fa-08ff-4b59-aa66-0346c1e882ff.jpg)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading Reader.Task.startRead and the reader's batchSize handling, focusing on the proposed incremental loop and its treatment of the final partial batch. Reproduce the reported import behavior, then define completion as incremental rows—including the latest incomplete batch—being delivered to the target database reliably.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.