hdfs write慢的问题
- Dominant language
- Java
- Stars
- 17.4k
- Forks
- 5.7k
- PR merge metrics
- No merged PRs in 30d
Description
从mysql -> hdfs,如果hdfs节点很慢,会出现mysql reader会出现下面的错误
om.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure The last packet successfully received from the server was 11,964 milliseconds ago. The last packet sent successfully to the server was 250,561 milliseconds ago
因为是全表导入,一般需要花2个小时时间,如果正好遇到比较差的datanode,就会出现上述错误,造成任务重试
请问一下,有什么优雅的处理方式,多谢
Contributor guide
No contributing guide indexed for this repository
Research direction
The issue describes a MySQL-to-HDFS full-table import that fails when an HDFS node is slow, but names no files or tests. Start by tracing the MySQL reader and HDFS writer paths and the JDBC communications-timeout handling. Done should be defined as a long import remaining reliable despite slow HDFS nodes, with coverage for the reported failure scenario.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, mysql
- Domain
- data-engineering, databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100