alibaba / alibaba/DataX

HdfsReader-OrcFile若单个文件大于256MB或者集群hdfs-block默认大小,会出现丢失数据

Open
#527 3 comments 2 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
17.4k
Forks
5.7k
PR merge metrics
No merged PRs in 30d

Description

HdfsReader DFSUtil类orcFileStartRead方法中如下代码有问题:

OrcSerde serde = new OrcSerde();
serde.initialize(conf, p);
StructObjectInspector inspector = (StructObjectInspector) serde.getObjectInspector();
InputFormat in = new OrcInputFormat();
FileInputFormat.setInputPaths(conf, orcFilePath.toString());
//If the network disconnected, will retry 45 times, each time the retry interval for 20 seconds
//Each file as a split
//TODO multy threads
InputSplit[] splits = in.getSplits(conf, 1);
RecordReader reader = in.getRecordReader(splits[0], conf, Reporter.NULL);
Object key = reader.createKey();
Object value = reader.createValue();

在执行这一段代码的时候,InputSplit切出来的数据块数量不一定是期望的1,而可能或更大
InputSplit[] splits = in.getSplits(conf, 1);
我测试的数据文件单个orc文件为671MB, 集群HADOOP-2.7.0单个数据块为256MB,需要切分为3块,而RecordReader reader = in.getRecordReader(splits[0], conf, Reporter.NULL); 只取了第一块,故这个才是数据丢失的原因!!!

望修读这个bug,这段代码上面还打着TODO,小坑~~

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in the DFSUtil class at the orcFileStartRead method and inspect how InputSplit[] splits is produced and how RecordReader is created from splits[0]. Reproduce the case with a 671MB ORC file and 256MB HDFS blocks; done means records from every split are retained rather than only the first block.

Written by the indexing model from the issue text.

Assessment

Tech stack
hadoop, java
Domain
data-engineering, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.