aws-samples / aws-samples/cql-replicator
Dictionary based compression support
- Dominant language
- Scala
- Stars
- 17
- Forks
- 14
- PR merge metrics
- No merged PRs in 30d
Description
**Is your feature request related to a problem? Please describe.**
Today the only supported compression algorithm is lz4. The feature request is to also support zstd with a dictionary.
**Describe the solution you'd like**
Allow choosing whether the compress data with lz4 or zstd with a dictionary. Also allow choosing the compression level.
Use a discovery job during the first run to generate a binary dictionary, which then will be uploaded to S3. Once
uploaded, replicators will leverage it for compression. Additionally, the customer can reuse the binary dictionary prepared by discovery for their application.
json-mapping:
"compressionConfig": {
"enabled": true,
"compression": zstd-dict-based/lz4,
"compressionLevel": 3,
"compressAllNonPrimaryColumns": false,
"compressNonPrimaryColumns": ["column-to-be-compressed"],
"targetNameColumn": "column-to-be-compressed"
},
The dictionary file will be ks_name_table_name_dictionary.json
**Describe alternatives you've considered**
The customer could provide the pretrained dictionary, but the ability to provide sample data instead should enable them to be more flexible and try out different compressions.
**Additional context**
Using a dictionary based compression is useful for columns with repeating values, for example low cardinality column, or a json column with similar field names in all/many rows.
Contributor guide
Assessment
This issue has not been assessed yet.