sqlserver到hdfs上,数据采集量不大可是执行了3个小时为什么
- Dominant language
- Java
- Stars
- 17.4k
- Forks
- 5.7k
- PR merge metrics
- No merged PRs in 30d
Description
同一个sqlserver库下的 ,其他表都没事 ,就是有2个表,是非常非常慢,数据量不大(4MB),我是按照通过sql语句一天的数据去采集。一直会这样 这个是10万条数据跑了(15MB) 3个小时
` 103072 records, 11801487 bytes | Speed 736B/s, 6 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.040s | All Task WaitReaderTime 12,695.251s | Percentage 0.00%
[INFO] 2020-01-07 03:39:51.838 cn.escheduler.server.worker.log.TaskLogger:[178] - [taskAppId=TASK_8_1332_11243] -> 2020-01-07 03:39:51.789 [job-0] INFO StandAloneJobContainerCommunicator - Total 103232 records, 11819764 bytes | Speed 913B/s, 8 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.040s | All Task WaitReaderTime 12,716.122s | Percentage 0.00%
[INFO] 2020-01-07 03:40:01.840 cn.escheduler.server.worker.log.TaskLogger:[178] - [taskAppId=TASK_8_1332_11243] -> 2020-01-07 03:40:01.790 [job-0] INFO StandAloneJobContainerCommunicator - Total 103328 records, 11830751 bytes | Speed 1.07KB/s, 9 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.040s | All Task WaitReaderTime 12,728.127s | Percentage 0.00%
[INFO] 2020-01-07 03:40:11.840 cn.escheduler.server.worker.log.TaskLogger:[178] - [taskAppId=TASK_8_1332_11243] -> 2020-01-07 03:40:11.791 [job-0] INFO StandAloneJobContainerCommunicator - Total 103424 records, 11841735 bytes | Speed 1.07KB/s, 9 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.040s | All Task WaitReaderTime 12,737.087s | Percentage 0.00%
[INFO] 2020-01-07 03:40:31.840 cn.escheduler.server.worker.log.TaskLogger:[178] - [taskAppId=TASK_8_1332_11243] -> 2020-01-07 03:40:31.792 [job-0] INFO StandAloneJobContainerCommunicator - Total 103584 records, 11860064 bytes | Speed 916B/s, 8 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.040s | All Task WaitReaderTime 12,757.017s | Percentage 0.00%
[INFO] 2020-01-07 03:40:41.842 cn.escheduler.server.worker.log.TaskLogger:[178] - [taskAppId=TASK_8_1332_11243] -> 2020-01-07 03:40:31.792 [job-0] INFO VMInfo -
[delta cpu info] =>
curDeltaCpu | averageCpu | maxDeltaCpu | minDeltaCpu
-1.00% | -1.00% | -1.00% | -1.00%
[delta memory info] =>
NAME | used_size | used_percent | max_used_size | max_percent
PS Eden Space | 115.01MB | 37.28% | 300.94MB | 99.84%
Code Cache | 15.14MB | 98.89% | 15.14MB | 99.31%
Compressed Class Space | 3.43MB | 94.76% | 3.43MB | 96.02%
PS Survivor Space | 2.99MB | 18.68% | 10.74MB | 25.28%
PS Old Gen | 14.65MB | 2.15% | 14.65MB | 2.15%
Metaspace | 31.46MB | 97.93% | 31.46MB | 98.41%
[delta gc info] =>
NAME | curDeltaGCCount | totalGCCount | maxDeltaGCCount | minDeltaGCCount | curDeltaGCTime | totalGCTime | maxDeltaGCTime | minDeltaGCTime
PS MarkSweep | 0 | 1 | 1 | 0 | 0.000s | 0.045s | 0.045s | 0.000s
PS Scavenge | 0 | 9 | 1 | 0 | 0.000s | 0.153s | 0.042s | 0.000s `
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the SQLServer-to-HDFS task configuration and the supplied DataX worker logs, comparing the two slow tables with tables that complete normally. Check the daily SQL scope and the WaitReaderTime metrics; done means identifying a reproducible cause and documenting a concrete fix or required configuration change.
Written by the indexing model from the issue text.
Assessment
- Domain
- data-engineering, databases, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100