alibaba / alibaba/DataX

并发数据计算疑问

Open
#846 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
17.4k
Forks
5.7k
PR merge metrics
No merged PRs in 30d

Description

举例来说,用户提交了一个DataX作业,并且配置了20个并发,目的是将一个100张分表的mysql数据同步到odps里面。 DataX的调度决策思路是:

DataXJob根据分库分表切分成了100个Task。
根据20个并发,DataX计算共需要分配4个TaskGroup。
4个TaskGroup平分切分好的100个Task,每一个TaskGroup负责以5个并发共计运行25个Task。

问题: 1.配置的20个并发是指单个TaskGroup的并发数吗? 如果是那么100/20=5应该是需要5个TaskGroup,这个4是怎么算的呢?
2.“每一个TaskGroup负责以5个并发共计运行25个Task”这句中的5个并发又是指定什么并发?
3.json的配置文件中,是由channel的个数来配置TaskGroup吗?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the DataXJob task-splitting and TaskGroup scheduling flow, then inspect how channel is read from the JSON configuration. Trace the meanings of the 20, 5, and 25 values in the issue's example. Done means documenting clear answers to all three concurrency questions.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, mysql
Domain
data-engineering, databases
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.