alibaba / alibaba/DataX

解决了DataX读取HDFS文件为空文件出现异常,代码未处理捕获异常

Open
#1,503 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
17.4k
Forks
5.7k
PR merge metrics
No merged PRs in 30d

Description

您好,在DataX开源版最新代码中,hdfsreader模块中DFSUtil类下的getHDFSAllFilesNORegex方法,在读取到空文件时会出现异常。该方法中在遍历目录时,遇到文件并不会判断文件长度,直接读取会导致程序异常退出,可以在读取文件时增加文件长度校验。修改后的代码如下:
private HashSet getHDFSAllFilesNORegex(String path, FileSystem hdfs) throws IOException {

// 获取要读取的文件的根目录
Path listFiles = new Path(path);

// If the network disconnected, this method will retry 45 times
// each time the retry interval for 20 seconds
// 获取要读取的文件的根目录的所有二级子文件目录
FileStatus stats[] = hdfs.listStatus(listFiles);

for (FileStatus f : stats) {
// 判断是不是目录,如果是目录,递归调用
if (f.isDirectory()) {
LOG.info(String.format("[%s] 是目录, 递归获取该目录下的文件", f.getPath().toString()));
getHDFSAllFilesNORegex(f.getPath().toString(), hdfs);
} else if (f.isFile()) {
if (f.getLen() != 0) {
addSourceFileByType(f.getPath().toString());
}
} else {
String message = String.format("该路径[%s]文件类型既不是目录也不是文件,插件自动忽略。",
f.getPath().toString());
LOG.info(message);
}
}
return sourceHDFSAllFilesList;
}
望采纳

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in the hdfsreader module at the DFSUtil.getHDFSAllFilesNORegex method described in the issue. Review how directory entries are handled and verify the behavior for zero-length files. Done means empty HDFS files are ignored without causing the reader to exit with an exception.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.