alibaba / alibaba/DataX

hdfs write慢的问题

Open
#502 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
17.4k
Forks
5.7k
PR merge metrics
No merged PRs in 30d

Description

从mysql -> hdfs,如果hdfs节点很慢,会出现mysql reader会出现下面的错误
om.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure The last packet successfully received from the server was 11,964 milliseconds ago. The last packet sent successfully to the server was 250,561 milliseconds ago

因为是全表导入,一般需要花2个小时时间,如果正好遇到比较差的datanode,就会出现上述错误,造成任务重试
请问一下,有什么优雅的处理方式,多谢

Contributor guide

No contributing guide indexed for this repository

Research direction

The issue describes a MySQL-to-HDFS full-table import that fails when an HDFS node is slow, but names no files or tests. Start by tracing the MySQL reader and HDFS writer paths and the JDBC communications-timeout handling. Done should be defined as a long import remaining reliable despite slow HDFS nodes, with coverage for the reported failure scenario.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, mysql
Domain
data-engineering, databases, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.