Separate dictionary references to enable dictionary usage for any combination of window size and content size
- Dominant language
- C
- Stars
- 27.9k
- Forks
- 2.6k
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 8
Description
_(Couldn't find an existing issue covering this, but happy to redirect conversation elsewhere.)_
My understanding based on @Cyan4973's [comment](https://github.com/httpwg/http-extensions/issues/2754#issuecomment-1986958132) in https://github.com/httpwg/http-extensions/issues/2754 is that Zstd stops being able to use a dictionary once we're more than a single window size "away" from the beginning of the content. In other words, the dictionary is almost "prepended" to the content for the purposes of being reference-able.
It seems like this entangles the ability to support backreferences based on some window size with the ability to use a dictionary at all when the window size is less than the size of the content itself.
In a browser context (and on mobile devices), we cannot use significantly large window sizes due to memory usage constraints.
Today, my understanding is that this means we will not benefit from dictionary usage beyond the first window-size number of bytes in the content.
Would it be possible to have a separate mechanism for referring to dictionary content that wasn't dependent on the window size? We're essentially already committing to having the dictionary in memory, and need a way to limit the maximum size of those dictionaries, so it seems quite unfortunate to not be able to benefit from having spent those resources to improve [de]compression.
Contributor guide
Assessment
This issue has not been assessed yet.