HdfsReader-OrcFile若单个文件大于256MB或者集群hdfs-block默认大小,会出现丢失数据
- Dominant language
- Java
- Stars
- 17.4k
- Forks
- 5.7k
- PR merge metrics
- No merged PRs in 30d
Description
HdfsReader DFSUtil类orcFileStartRead方法中如下代码有问题:
OrcSerde serde = new OrcSerde();
serde.initialize(conf, p);
StructObjectInspector inspector = (StructObjectInspector) serde.getObjectInspector();
InputFormat in = new OrcInputFormat();
FileInputFormat.setInputPaths(conf, orcFilePath.toString());
//If the network disconnected, will retry 45 times, each time the retry interval for 20 seconds
//Each file as a split
//TODO multy threads
InputSplit[] splits = in.getSplits(conf, 1);
RecordReader reader = in.getRecordReader(splits[0], conf, Reporter.NULL);
Object key = reader.createKey();
Object value = reader.createValue();
在执行这一段代码的时候,InputSplit切出来的数据块数量不一定是期望的1,而可能或更大
InputSplit[] splits = in.getSplits(conf, 1);
我测试的数据文件单个orc文件为671MB, 集群HADOOP-2.7.0单个数据块为256MB,需要切分为3块,而RecordReader reader = in.getRecordReader(splits[0], conf, Reporter.NULL); 只取了第一块,故这个才是数据丢失的原因!!!
望修读这个bug,这段代码上面还打着TODO,小坑~~
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in the DFSUtil class at the orcFileStartRead method and inspect how InputSplit[] splits is produced and how RecordReader is created from splits[0]. Reproduce the case with a 671MB ORC file and 256MB HDFS blocks; done means records from every split are retained rather than only the first block.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- hadoop, java
- Domain
- data-engineering, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 45/100