HdfsWriter,orc格式文件设置压缩格式参数部分存在bug,最终都使用的ZLIB压缩输出
- Dominant language
- Java
- Stars
- 17.4k
- Forks
- 5.7k
- PR merge metrics
- No merged PRs in 30d
Description
HdfsHelper 类中的 orcFileStartWrite 方法中现在判断压缩格式后,设置了参数 "mapreduce.output.fileoutputformat.compress.codec","mapreduce.output.fileoutputformat.compress":
```java
if(!"NONE".equalsIgnoreCase(compress) && null != compress ) {
Class codecClass = getCompressCodec(compress);
if (null != codecClass) {
outFormat.setOutputCompressorClass(conf, codecClass);
}
}
```
但是 RecordWrite 设置压缩类型是最终读取的是 "hive.exec.orc.default.compress" 这个参数,该值默认为ZLIB。具体调用代码如下
``` java
// 所在目录 com.alibaba.datax.plugin.writer.hdfswriter.HdfsHelper
//outFormat 创建时的方法 FileOutputFormat outFormat = new OrcOutputFormat();
RecordWriter writer = outFormat.getRecordWriter(fileSystem, conf, fileName, Reporter.NULL);
```
```
//getRecordWriter 实际调用了 OrcOutputFormat类的 getRecordWriter,最终调用OrcFile类的WriterOptions 方法
//所在包目录 package org.apache.hadoop.hive.ql.io.orc; 类名为OrcFile
this.compressValue = CompressionKind.valueOf(HiveConf.getVar(conf, ConfVars.HIVE_ORC_DEFAULT_COMPRESS));
```
所以判断是否使用压缩的参数全都无效,一直使用的默认压缩ZLIB
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in com.alibaba.datax.plugin.writer.hdfswriter.HdfsHelper.orcFileStartWrite, then trace the OrcOutputFormat getRecordWriter call into org.apache.hadoop.hive.ql.io.orc.OrcFile.WriterOptions. Verify which configuration value controls the ORC compression and confirm that the requested compression is honored instead of always falling back to ZLIB.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- hadoop, java
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100