apache / apache/arrow-go

[Go][Parquet] Support fallback encoding

Open
#31 4 comments 0 reactions 0 assignees View on GitHub
Component: Parquet Type: enhancement
Dominant language
Assembly
Stars
404
Forks
145
Avg merge
2d 4h
Merged PRs (30d)
87

Description

### Describe the enhancement requested

[WithEncoding](https://github.com/apache/arrow/blob/21238a7f5ebb224410d84ecf32dbe3556cab363a/go/parquet/writer_properties.go#L192) comments are misleading users now.
```
// WithEncoding defines the encoding that is used when we aren't using dictionary encoding.
//
// This is either applied if dictionary encoding is disabled, or if we fallback if the dictionary
// grew too large.
func WithEncoding(encoding Encoding) WriterProperty {
return func(cfg *writerPropConfig) {
if encoding == Encodings.PlainDict || encoding == Encodings.RLEDict {
panic("parquet: can't use dictionary encoding as fallback encoding")
}
cfg.wr.defColumnProps.Encoding = encoding
}
}
```
Currently, Go parquet fallback encoding is hardcoded as plain encoding. See [fallbackToPlain](https://github.com/apache/arrow/blob/21238a7f5ebb224410d84ecf32dbe3556cab363a/go/parquet/file/column_writer.go#L67). It is better to support custom fallback encoding.

### Component(s)

Go, Parquet

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.