alibaba / alibaba/DataX

sqlserver到hdfs上,数据采集量不大可是执行了3个小时为什么

Open
#570 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
17.4k
Forks
5.7k
PR merge metrics
No merged PRs in 30d

Description

同一个sqlserver库下的 ,其他表都没事 ,就是有2个表,是非常非常慢,数据量不大(4MB),我是按照通过sql语句一天的数据去采集。一直会这样 这个是10万条数据跑了(15MB) 3个小时
` 103072 records, 11801487 bytes | Speed 736B/s, 6 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.040s | All Task WaitReaderTime 12,695.251s | Percentage 0.00%
[INFO] 2020-01-07 03:39:51.838 cn.escheduler.server.worker.log.TaskLogger:[178] - [taskAppId=TASK_8_1332_11243] -> 2020-01-07 03:39:51.789 [job-0] INFO StandAloneJobContainerCommunicator - Total 103232 records, 11819764 bytes | Speed 913B/s, 8 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.040s | All Task WaitReaderTime 12,716.122s | Percentage 0.00%
[INFO] 2020-01-07 03:40:01.840 cn.escheduler.server.worker.log.TaskLogger:[178] - [taskAppId=TASK_8_1332_11243] -> 2020-01-07 03:40:01.790 [job-0] INFO StandAloneJobContainerCommunicator - Total 103328 records, 11830751 bytes | Speed 1.07KB/s, 9 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.040s | All Task WaitReaderTime 12,728.127s | Percentage 0.00%
[INFO] 2020-01-07 03:40:11.840 cn.escheduler.server.worker.log.TaskLogger:[178] - [taskAppId=TASK_8_1332_11243] -> 2020-01-07 03:40:11.791 [job-0] INFO StandAloneJobContainerCommunicator - Total 103424 records, 11841735 bytes | Speed 1.07KB/s, 9 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.040s | All Task WaitReaderTime 12,737.087s | Percentage 0.00%
[INFO] 2020-01-07 03:40:31.840 cn.escheduler.server.worker.log.TaskLogger:[178] - [taskAppId=TASK_8_1332_11243] -> 2020-01-07 03:40:31.792 [job-0] INFO StandAloneJobContainerCommunicator - Total 103584 records, 11860064 bytes | Speed 916B/s, 8 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.040s | All Task WaitReaderTime 12,757.017s | Percentage 0.00%
[INFO] 2020-01-07 03:40:41.842 cn.escheduler.server.worker.log.TaskLogger:[178] - [taskAppId=TASK_8_1332_11243] -> 2020-01-07 03:40:31.792 [job-0] INFO VMInfo -
[delta cpu info] =>
curDeltaCpu | averageCpu | maxDeltaCpu | minDeltaCpu
-1.00% | -1.00% | -1.00% | -1.00%


[delta memory info] =>
NAME | used_size | used_percent | max_used_size | max_percent
PS Eden Space | 115.01MB | 37.28% | 300.94MB | 99.84%
Code Cache | 15.14MB | 98.89% | 15.14MB | 99.31%
Compressed Class Space | 3.43MB | 94.76% | 3.43MB | 96.02%
PS Survivor Space | 2.99MB | 18.68% | 10.74MB | 25.28%
PS Old Gen | 14.65MB | 2.15% | 14.65MB | 2.15%
Metaspace | 31.46MB | 97.93% | 31.46MB | 98.41%

[delta gc info] =>
NAME | curDeltaGCCount | totalGCCount | maxDeltaGCCount | minDeltaGCCount | curDeltaGCTime | totalGCTime | maxDeltaGCTime | minDeltaGCTime
PS MarkSweep | 0 | 1 | 1 | 0 | 0.000s | 0.045s | 0.045s | 0.000s
PS Scavenge | 0 | 9 | 1 | 0 | 0.000s | 0.153s | 0.042s | 0.000s `

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the SQLServer-to-HDFS task configuration and the supplied DataX worker logs, comparing the two slow tables with tables that complete normally. Check the daily SQL scope and the WaitReaderTime metrics; done means identifying a reproducible cause and documenting a concrete fix or required configuration change.

Written by the indexing model from the issue text.

Assessment

Domain
data-engineering, databases, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.