[C++] Util: Compression supports a Compression/Decompression Context
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the enhancement requested
In compression, zstd supports a Compression/Decompression context[1]. It will make it better for system memory usage. RocksDB can utilize the Context[2] and Context Cache[3].
So here I propose an support for context.
```c++
class ARROW_EXPORT CodecContext {
std::unique_ptr compress_ctx
std::unique_ptr decompress_ctx;
};
/// \brief Compression codec options
class ARROW_EXPORT CodecOptions {
public:
int compression_level;
+ std::unique_ptr context;
};
```
(this require compression / decompression not run concurrently, if they run concurrently, we may need another interface like
```
/// \brief One-shot decompression function
///
/// output_buffer_len must be correct and therefore be obtained in advance.
/// The actual decompressed length is returned.
///
/// \note One-shot decompression is not always compatible with streaming
/// compression. Depending on the codec (e.g. LZ4), different formats may
/// be used.
virtual Result Decompress(int64_t input_len, const uint8_t* input,
int64_t output_buffer_len,
uint8_t* output_buffer, CompressionCtx*) = 0;
/// \brief One-shot compression function
///
/// output_buffer_len must first have been computed using MaxCompressedLen().
/// The actual compressed length is returned.
///
/// \note One-shot compression is not always compatible with streaming
/// decompression. Depending on the codec (e.g. LZ4), different formats may
/// be used.
virtual Result Compress(int64_t input_len, const uint8_t* input,
int64_t output_buffer_len, uint8_t* output_buffer, DecompressionCtx*) = 0;
```
1. https://raw.githack.com/facebook/zstd/release/doc/zstd_manual.html
2. https://github.com/facebook/rocksdb/blob/a09c141dde51372d14bcfd3affdd242f1248c761/util/compression.h#L362
3. https://github.com/facebook/rocksdb/blob/a09c141dde51372d14bcfd3affdd242f1248c761/util/compression_context_cache.h#L27
### Component(s)
C++
Contributor guide
Assessment
This issue has not been assessed yet.