facebook / facebook/zstd

Separate dictionary references to enable dictionary usage for any combination of window size and content size

Open
#4,017 1 comment 0 reactions 1 assignee Claimed by @Cyan4973 View on GitHub
Dominant language
C
Stars
27.9k
Forks
2.6k
Avg merge
1d 3h
Merged PRs (30d)
8

Description

_(Couldn't find an existing issue covering this, but happy to redirect conversation elsewhere.)_

My understanding based on @Cyan4973's [comment](https://github.com/httpwg/http-extensions/issues/2754#issuecomment-1986958132) in https://github.com/httpwg/http-extensions/issues/2754 is that Zstd stops being able to use a dictionary once we're more than a single window size "away" from the beginning of the content. In other words, the dictionary is almost "prepended" to the content for the purposes of being reference-able.

It seems like this entangles the ability to support backreferences based on some window size with the ability to use a dictionary at all when the window size is less than the size of the content itself.

In a browser context (and on mobile devices), we cannot use significantly large window sizes due to memory usage constraints.

Today, my understanding is that this means we will not benefit from dictionary usage beyond the first window-size number of bytes in the content.

Would it be possible to have a separate mechanism for referring to dictionary content that wasn't dependent on the window size? We're essentially already committing to having the dictionary in memory, and need a way to limit the maximum size of those dictionaries, so it seems quite unfortunate to not be able to benefit from having spent those resources to improve [de]compression.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.