apache / apache/arrow

[python] Missing Compression Codecs in Schema.fbs

Open
#36,188 2 comments 0 reactions 0 assignees View on GitHub
Component: Documentation Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

In `format/Message.fbs`, the Arrow project documents two options for compression:

```
enum CompressionType:byte {
// LZ4 frame format, for portability, as provided by lz4frame.h or wrappers
// thereof. Not to be confused with "raw" (also called "block") format
// provided by lz4.h
LZ4_FRAME,

// Zstandard
ZSTD
}
```

However, [pyarrow.Codec](https://arrow.apache.org/docs/python/generated/pyarrow.Codec.html) suggests that there are many more options, at the least: gzip, bz2, brotli, lz4, lz4_frame, lz4_raw, zstd, snappy. And this is confirmed by [the 5.0.0 release notes](https://arrow.apache.org/blog/2021/07/29/5.0.0-release/):

> The new LZ4_RAW compression scheme was implemented (PARQUET-1998). Unlike the legacy LZ4 compression scheme, it is defined unambiguously and should provide better portability once other Parquet implementations catch up.

But the flatbuffers definition lags behind. I've looked through some of the generated cpp files, and I cannot figure out what the flatbuffers value for `LZ4_RAW` is supposed to be. I would greatly appreciate it if someone who understands the living spec could update the formal spec.

### Component(s)

Documentation

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.