facebook / facebook/zstd

Question about LZ77 decode dependencies and parallelization (learning)

Ouverte
#4,630 2 commentaires 0 réactions 0 personnes assignées Voir sur GitHub
Langage dominant
C
Étoiles
27.9k
Forks
2.6k
Merge moyen
1 j 3 h
PR mergées (30 j)
8

Description

Hi,

I’m not very experienced with compression, so sorry if this is a naive question.

I’ve been experimenting with an LZ77-like approach and got a result that I don’t fully understand.

I managed to get:
- bit-perfect decoding
- ~10 GB/s decode on a ~1GB file
- while still using a global compression context

The way I structured it is roughly:

- compression is global (matches can reference earlier data across the whole input)
- but I split the output into blocks
- and precompute offsets so each block knows where its literals / offsets / lengths / commands are

So blocks are not independently compressed, but I try to make them independently decodable.

This seems to allow parallel decode, which is where the speed comes from.

But I feel like I’m probably misunderstanding something fundamental, because I don’t see this pattern used much.

My questions:

- Is there a fundamental reason why LZ77 decode is usually strictly sequential?
- Are there known designs that separate compression context from decode dependency like this?
- What are the main pitfalls of this approach in real implementations?

I’m not claiming this is new — I’m just trying to understand what I might be missing.

Happy to share code or more details if that helps.

Thanks

Guide de contribution

Ouvrir le guide de contribution

Évaluation

Cette issue n'a pas encore été évaluée.

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.