apache / apache/arrow

[C++] Support reading Hadoop-snappy File Format Directly

Open
#36,608 13 comments 0 reactions 0 assignees View on GitHub
Component: C++ Type: enhancement
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the enhancement requested

Hadoop-snappy File Format is widely used for BigData and ML frameworks; We use spark or hadoop mr to preprocess data for furture processing such like Data warehouse and DeepLearning training;

Hadoop-snappy File is a compressed file consists of one or more blocks. A block consists of uncompressed length (big endian 4 byte integer) and one or more subblocks.

Arrow could read a file which is fully compressed with Snappy Codec but fails to read Hadoop-snappy File Format;

If arrow can read the Hadoop-snappy File consisted of a serials of chunks, we can use the output from Spark directly and save the HDFS storage cost. All directly reading Haoop-snappy file will greatly extend the Arrow's using scenarios~

Within bytedance.com, almost 40% files are compressed by Snappy and stored in HDFS filesystem; we use arrow to read Hadoop-snappy File and Hadoop-Zstd file which save computing resources greatly~

### Component(s)

C++

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.