alibaba / alibaba/DataX

hdfsreader组件中有一处输入流未关闭导致大量连接CLOSE_WAIT,最终Address already in use

Open
#1,047 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Java
Stars
17.4k
Forks
5.7k
PR merge metrics
No merged PRs in 30d

Description

可能是我使用方式的问题,将datax作为一个jar内嵌到了项目中,生产过程中报出
![image](https://user-images.githubusercontent.com/27263174/122363581-a3c60700-cf8b-11eb-8839-2cde1fc50e4e.png)

通过`netstat -anpt | wc -l`查看客户端连接datanode:1004端口状态是CLOSE_WAIT的已经将近20w了
查看代码发现hdfsreader组件中**com.alibaba.datax.plugin.reader.hdfsreader.DFSUtil类中的checkHdfsFileType方法**中FSDataInputStream in = fs.open(file)在项目中未关闭,未关闭的流导致大量连接等待关闭占用了大量端口,导致新任务失败,找个合适的位置in.close()就可以了

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in com.alibaba.datax.plugin.reader.hdfsreader.DFSUtil, at the checkHdfsFileType method, and inspect how the FSDataInputStream from fs.open(file) is handled. Confirm the stream is closed on the relevant paths, then verify that HDFS reader jobs no longer leave large numbers of Datanode connections in CLOSE_WAIT.

Written by the indexing model from the issue text.

Assessment

Tech stack
hadoop, java
Domain
data-engineering, distributed-systems
Issue type
Bug
Difficulty
1/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.