facebook / facebook/zstd

Question about LZ77 decode dependencies and parallelization (learning)

Aperta
#4,630 2 commenti 0 reazioni 0 assegnatari Vedi su GitHub
Lingua principale
C
Stelle
27.9k
Fork
2.6k
Merge medio
1g 3h
PR unite (30g)
8

Descrizione

Hi,

I’m not very experienced with compression, so sorry if this is a naive question.

I’ve been experimenting with an LZ77-like approach and got a result that I don’t fully understand.

I managed to get:
- bit-perfect decoding
- ~10 GB/s decode on a ~1GB file
- while still using a global compression context

The way I structured it is roughly:

- compression is global (matches can reference earlier data across the whole input)
- but I split the output into blocks
- and precompute offsets so each block knows where its literals / offsets / lengths / commands are

So blocks are not independently compressed, but I try to make them independently decodable.

This seems to allow parallel decode, which is where the speed comes from.

But I feel like I’m probably misunderstanding something fundamental, because I don’t see this pattern used much.

My questions:

- Is there a fundamental reason why LZ77 decode is usually strictly sequential?
- Are there known designs that separate compression context from decode dependency like this?
- What are the main pitfalls of this approach in real implementations?

I’m not claiming this is new — I’m just trying to understand what I might be missing.

Happy to share code or more details if that helps.

Thanks

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.