alibaba / alibaba/DataX

splitPk后,生成重复数据文件

Open
#2,078 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
17.4k
Forks
5.7k
PR merge metrics
No merged PRs in 30d

Description

读插件为oceanbasev10reader,写插件为txtfilewriter
不加splitPk参数,仅生成一个文件。加上splitPk后,生成了多个文件,但是核查发现,每个文件的内容都一模一样,都是全量数据。
确认了splitPk的字段是递增且唯一的,请问该如何保证生成的多个文件应该为实际拆分任务查询出来的数据,而不是全量数据呢?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the case with oceanbasev10reader, txtfilewriter, and splitPk, comparing the generated files with and without splitPk. Trace how each split task builds and executes its query; done means multiple output files contain their respective task rows rather than identical full datasets.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.