alibaba / alibaba/DataX

HdfsWriter,orc格式文件设置压缩格式参数部分存在bug,最终都使用的ZLIB压缩输出

Open
#472 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
17.4k
Forks
5.7k
PR merge metrics
No merged PRs in 30d

Description

HdfsHelper 类中的 orcFileStartWrite 方法中现在判断压缩格式后,设置了参数 "mapreduce.output.fileoutputformat.compress.codec","mapreduce.output.fileoutputformat.compress":
```java
if(!"NONE".equalsIgnoreCase(compress) && null != compress ) {
Class codecClass = getCompressCodec(compress);
if (null != codecClass) {
outFormat.setOutputCompressorClass(conf, codecClass);
}
}
```
但是 RecordWrite 设置压缩类型是最终读取的是 "hive.exec.orc.default.compress" 这个参数,该值默认为ZLIB。具体调用代码如下
``` java
// 所在目录 com.alibaba.datax.plugin.writer.hdfswriter.HdfsHelper
//outFormat 创建时的方法 FileOutputFormat outFormat = new OrcOutputFormat();
RecordWriter writer = outFormat.getRecordWriter(fileSystem, conf, fileName, Reporter.NULL);
```
```
//getRecordWriter 实际调用了 OrcOutputFormat类的 getRecordWriter,最终调用OrcFile类的WriterOptions 方法
//所在包目录 package org.apache.hadoop.hive.ql.io.orc; 类名为OrcFile
this.compressValue = CompressionKind.valueOf(HiveConf.getVar(conf, ConfVars.HIVE_ORC_DEFAULT_COMPRESS));
```

所以判断是否使用压缩的参数全都无效,一直使用的默认压缩ZLIB

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in com.alibaba.datax.plugin.writer.hdfswriter.HdfsHelper.orcFileStartWrite, then trace the OrcOutputFormat getRecordWriter call into org.apache.hadoop.hive.ql.io.orc.OrcFile.WriterOptions. Verify which configuration value controls the ORC compression and confirm that the requested compression is honored instead of always falling back to ZLIB.

Written by the indexing model from the issue text.

Assessment

Tech stack
hadoop, java
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.