hdfsreader组件中有一处输入流未关闭导致大量连接CLOSE_WAIT,最终Address already in use
- Dominant language
- Java
- Stars
- 17.4k
- Forks
- 5.7k
- PR merge metrics
- No merged PRs in 30d
Description
可能是我使用方式的问题,将datax作为一个jar内嵌到了项目中,生产过程中报出

通过`netstat -anpt | wc -l`查看客户端连接datanode:1004端口状态是CLOSE_WAIT的已经将近20w了
查看代码发现hdfsreader组件中**com.alibaba.datax.plugin.reader.hdfsreader.DFSUtil类中的checkHdfsFileType方法**中FSDataInputStream in = fs.open(file)在项目中未关闭,未关闭的流导致大量连接等待关闭占用了大量端口,导致新任务失败,找个合适的位置in.close()就可以了
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in com.alibaba.datax.plugin.reader.hdfsreader.DFSUtil, at the checkHdfsFileType method, and inspect how the FSDataInputStream from fs.open(file) is handled. Confirm the stream is closed on the relevant paths, then verify that HDFS reader jobs no longer leave large numbers of Datanode connections in CLOSE_WAIT.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- hadoop, java
- Domain
- data-engineering, distributed-systems
- Issue type
- Bug
- Difficulty
- 1/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 55/100