alibaba / alibaba/DataX

并发时 Communication 统计有异常

Open
#1,065 15 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
17.4k
Forks
5.7k
PR merge metrics
No merged PRs in 30d

Description

在项目实践中同时触发两个job任务 datax 打印出日志任务
2021-07-14 19:56:22.531 INFO 14772 --- [job-26] com.alibaba.datax.core.job.JobContainer :
任务启动时刻 : 2021-07-14 19:56:12
任务结束时刻 : 2021-07-14 19:56:22
任务总计耗时 : 10s
任务平均流量 : 2.49KB/s
记录写入速度 : 17rec/s
读出记录总数 : 170
读写失败总数 : 0

: Total 170 records, 25495 bytes | Speed 2.49KB/s, 17 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.000s | All Task WaitReaderTime 0.000s | Percentage 100.00%
2021-07-14 19:56:22.763 INFO 14772 --- [job-852] com.alibaba.datax.core.job.JobContainer :
任务启动时刻 : 2021-07-14 19:56:12
任务结束时刻 : 2021-07-14 19:56:22
任务总计耗时 : 10s
任务平均流量 : 2.49KB/s
记录写入速度 : 17rec/s
读出记录总数 : 170
读写失败总数 : 0
其中第一个任务真实交换量是170条数据 第二个任务的交换量不是170条数据。

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at com.alibaba.datax.core.job.JobContainer and inspect how Communication statistics are collected or reported when two jobs run concurrently. Reproduce the reported two-job scenario and compare each job's record totals; done means each job's log reports its own exchange count rather than sharing another job's value.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.