DataX支持增量同步吗?我想改造Reader.Task的startRead方法完成增量同步好像会出现尾部的一部分数据无法导入
- Dominant language
- Java
- Stars
- 17.4k
- Forks
- 5.7k
- PR merge metrics
- No merged PRs in 30d
Description
想自己改一下源码支持增量同步 改造源码的如下,就是通过在reader内通过一个死循环,并且使用递增的id作为条件查询最新的数据,但是目前有个问题就是最新的一批数据会出现无法导入目标数据库的情况 个人猜测是因为batchSize没有达到导致的,请问有什么办法解决这个问题吗,就是在reader的这个循环内调用一下类似于IO流中flush() 这样的方法手动让这部分导入目标数据库

Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading Reader.Task.startRead and the reader's batchSize handling, focusing on the proposed incremental loop and its treatment of the final partial batch. Reproduce the reported import behavior, then define completion as incremental rows—including the latest incomplete batch—being delivered to the target database reliably.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100