[C++] Support reading Hadoop-snappy File Format Directly
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the enhancement requested
Hadoop-snappy File Format is widely used for BigData and ML frameworks; We use spark or hadoop mr to preprocess data for furture processing such like Data warehouse and DeepLearning training;
Hadoop-snappy File is a compressed file consists of one or more blocks. A block consists of uncompressed length (big endian 4 byte integer) and one or more subblocks.
Arrow could read a file which is fully compressed with Snappy Codec but fails to read Hadoop-snappy File Format;
If arrow can read the Hadoop-snappy File consisted of a serials of chunks, we can use the output from Spark directly and save the HDFS storage cost. All directly reading Haoop-snappy file will greatly extend the Arrow's using scenarios~
Within bytedance.com, almost 40% files are compressed by Snappy and stored in HDFS filesystem; we use arrow to read Hadoop-snappy File and Hadoop-Zstd file which save computing resources greatly~
### Component(s)
C++
Contributor guide
Assessment
This issue has not been assessed yet.