[Feature Request]: Support Valkey GLIDE decompression
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 8.8k
- Forks
- 491
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 49
Description
Problem description
I'm using Valkey GLIDE as my client to talk to a Valkey cluster. They have an (experimental) feature to enable compression https://glide.valkey.io/concepts/client-features/compression/, with either ZSTD or LZ4.
I thought that this would work great with the Enable Automatic Data Decompression -> ZSTD option. But it doesn't.
The reason is that GLIDE does this;
- A short magic header is prepended to identify the data as compressed and record which backend was used.
Which is defined here https://github.com/valkey-io/valkey-glide/blob/d3feeaaf08e0cb4b8eff17ea7df1ddbc6e6d0c73/glide-core/src/compression.rs#L1103
So a 5 byte magic header is prepended to the actual compressed data. Breaking the automatic compression detection in Redis Insight.
Once those 5 bytes are removed, decompression works as expected. I have tested this on the dev build against my ZSTD compressed data and it works. I have not tested against LZ4 which seems to also have 4 bytes representing the original size https://github.com/valkey-io/valkey-glide/blob/d3feeaaf08e0cb4b8eff17ea7df1ddbc6e6d0c73/glide-core/src/compression.rs?#L774-L777 before the actual data.
I'm thinking that the ZSTD | LZ4 could also check for these initial magic bytes and unpack before handing the data to the decompression fn. Or that a new option is introduced in the dropdown. Maybe just Valkey GLIDE? Since they encode in their envelope which backend was used. Which would then shift the data until we arrive att the compressed bytes and then hand them over to the existing decompression path.
Is this something you'd be interested in supporting? If so, any preference on how? I could take a look at implementing a solution if you want.
How often do you encounter this problem Every time I view data compressed by GLIDE.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing Redis Insight's automatic decompression option and compare its detection with the compression envelope in glide-core/src/compression.rs, especially the linked ZSTD and LZ4 sections. Validate the behavior against GLIDE-compressed ZSTD data and then LZ4 data; done means the GLIDE header is handled before the existing decompression path.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- redis, typescript
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100