aws-samples / aws-samples/cql-replicator

Dictionary based compression support

Open
#212 0 comments 0 reactions 1 assignee Claimed by @nwheeler81 View on GitHub
enhancement
Dominant language
Scala
Stars
17
Forks
14
PR merge metrics
No merged PRs in 30d

Description

**Is your feature request related to a problem? Please describe.**
Today the only supported compression algorithm is lz4. The feature request is to also support zstd with a dictionary.

**Describe the solution you'd like**
Allow choosing whether the compress data with lz4 or zstd with a dictionary. Also allow choosing the compression level.

Use a discovery job during the first run to generate a binary dictionary, which then will be uploaded to S3. Once
uploaded, replicators will leverage it for compression. Additionally, the customer can reuse the binary dictionary prepared by discovery for their application.

json-mapping:
"compressionConfig": {
"enabled": true,
"compression": zstd-dict-based/lz4,
"compressionLevel": 3,
"compressAllNonPrimaryColumns": false,
"compressNonPrimaryColumns": ["column-to-be-compressed"],
"targetNameColumn": "column-to-be-compressed"
},

The dictionary file will be ks_name_table_name_dictionary.json

**Describe alternatives you've considered**
The customer could provide the pretrained dictionary, but the ability to provide sample data instead should enable them to be more flexible and try out different compressions.

**Additional context**
Using a dictionary based compression is useful for columns with repeating values, for example low cardinality column, or a json column with similar field names in all/many rows.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.