apache / apache/arrow

[C++] Util: Compression supports a Compression/Decompression Context

Open
#37,169 18 comments 0 reactions 0 assignees View on GitHub
Component: C++ Type: enhancement
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the enhancement requested

In compression, zstd supports a Compression/Decompression context[1]. It will make it better for system memory usage. RocksDB can utilize the Context[2] and Context Cache[3].

So here I propose an support for context.

```c++
class ARROW_EXPORT CodecContext {
std::unique_ptr compress_ctx
std::unique_ptr decompress_ctx;
};

/// \brief Compression codec options
class ARROW_EXPORT CodecOptions {
public:
int compression_level;
+ std::unique_ptr context;
};
```

(this require compression / decompression not run concurrently, if they run concurrently, we may need another interface like

```
/// \brief One-shot decompression function
///
/// output_buffer_len must be correct and therefore be obtained in advance.
/// The actual decompressed length is returned.
///
/// \note One-shot decompression is not always compatible with streaming
/// compression. Depending on the codec (e.g. LZ4), different formats may
/// be used.
virtual Result Decompress(int64_t input_len, const uint8_t* input,
int64_t output_buffer_len,
uint8_t* output_buffer, CompressionCtx*) = 0;

/// \brief One-shot compression function
///
/// output_buffer_len must first have been computed using MaxCompressedLen().
/// The actual compressed length is returned.
///
/// \note One-shot compression is not always compatible with streaming
/// decompression. Depending on the codec (e.g. LZ4), different formats may
/// be used.
virtual Result Compress(int64_t input_len, const uint8_t* input,
int64_t output_buffer_len, uint8_t* output_buffer, DecompressionCtx*) = 0;
```

1. https://raw.githack.com/facebook/zstd/release/doc/zstd_manual.html
2. https://github.com/facebook/rocksdb/blob/a09c141dde51372d14bcfd3affdd242f1248c761/util/compression.h#L362
3. https://github.com/facebook/rocksdb/blob/a09c141dde51372d14bcfd3affdd242f1248c761/util/compression_context_cache.h#L27

### Component(s)

C++

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.