alibaba / alibaba/DataX

datax_v202309并没有实现 HdfsReader/HdfsWriter支持parquet读写能力

Open
#2,120 5 comments 1 reaction 0 assignees View on GitHub
Dominant language
Java
Stars
17.4k
Forks
5.7k
PR merge metrics
No merged PRs in 30d

Description

我下了datax_v202309,设置hdfsreader的 "fileType": "parquet",运行的时候报错:
![image](https://github.com/alibaba/DataX/assets/10126251/b3a4b25f-b542-4b14-8926-630f9f87bdf6)
检查代码发现很多地方实际上没有支持parquet格式,如:
com.alibaba.datax.plugin.reader.hdfsreader.DFSUtil#checkHdfsFileType

public boolean checkHdfsFileType(String filepath, String specifiedFileType) {

Path file = new Path(filepath);

try {
FileSystem fs = FileSystem.get(hadoopConf);
FSDataInputStream in = fs.open(file);

if (StringUtils.equalsIgnoreCase(specifiedFileType, Constant.CSV)
|| StringUtils.equalsIgnoreCase(specifiedFileType, Constant.TEXT)) {

boolean isORC = isORCFile(file, fs, in);// 判断是否是 ORC File
if (isORC) {
return false;
}
boolean isRC = isRCFile(filepath, in);// 判断是否是 RC File
if (isRC) {
return false;
}
boolean isSEQ = isSequenceFile(filepath, in);// 判断是否是 Sequence File
if (isSEQ) {
return false;
}
// 如果不是ORC,RC和SEQ,则默认为是TEXT或CSV类型
return !isORC && !isRC && !isSEQ;

} else if (StringUtils.equalsIgnoreCase(specifiedFileType, Constant.ORC)) {

return isORCFile(file, fs, in);
} else if (StringUtils.equalsIgnoreCase(specifiedFileType, Constant.RC)) {

return isRCFile(filepath, in);
} else if (StringUtils.equalsIgnoreCase(specifiedFileType, Constant.SEQ)) {

return isSequenceFile(filepath, in);
}

} catch (Exception e) {
String message = String.format("检查文件[%s]类型失败,目前支持ORC,SEQUENCE,RCFile,TEXT,CSV五种格式的文件," +
"请检查您文件类型和文件是否正确。", filepath);
LOG.error(message);
throw DataXException.asDataXException(HdfsReaderErrorCode.READ_FILE_ERROR, message, e);
}
return false;
}

这里面还是没有支持parquet格式

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by inspecting HdfsReader and HdfsWriter, especially com.alibaba.datax.plugin.reader.hdfsreader.DFSUtil#checkHdfsFileType and the referenced file-type constants. Trace the existing ORC, RCFile, and SequenceFile handling to identify the remaining parquet gaps. Done means parquet read and write support works and the file-type validation no longer rejects valid parquet files.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.