facebook / facebook/zstd

Question about LZ77 decode dependencies and parallelization (learning)

オープン
#4,630 コメント 2 件 リアクション 0 件 担当者 0 名 GitHub で見る
主要言語
C
スター
27.9k
フォーク
2.6k
平均マージ
1日 3時間
マージ済み PR(30日)
8

説明

Hi,

I’m not very experienced with compression, so sorry if this is a naive question.

I’ve been experimenting with an LZ77-like approach and got a result that I don’t fully understand.

I managed to get:
- bit-perfect decoding
- ~10 GB/s decode on a ~1GB file
- while still using a global compression context

The way I structured it is roughly:

- compression is global (matches can reference earlier data across the whole input)
- but I split the output into blocks
- and precompute offsets so each block knows where its literals / offsets / lengths / commands are

So blocks are not independently compressed, but I try to make them independently decodable.

This seems to allow parallel decode, which is where the speed comes from.

But I feel like I’m probably misunderstanding something fundamental, because I don’t see this pattern used much.

My questions:

- Is there a fundamental reason why LZ77 decode is usually strictly sequential?
- Are there known designs that separate compression context from decode dependency like this?
- What are the main pitfalls of this approach in real implementations?

I’m not claiming this is new — I’m just trying to understand what I might be missing.

Happy to share code or more details if that helps.

Thanks

コントリビューションガイド

コントリビューションガイドを開く

評価

この issue はまだ評価されていません。

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。