HTTP custom dictionary auto discovery
- 主要言語
- C
- スター
- 27.9k
- フォーク
- 2.6k
- 平均マージ
- 1日 3時間
- マージ済み PR(30日)
- 8
説明
**Is your feature request related to a problem? Please describe.**
Zstd would be awesome if implemented in browsers with standard dictionaries, and from what I've read, there are efforts to craft such standard dictionaries for common web formats that would provide some better compression for typical cases.
However, I think nothing would beat a custom dictionary trained on a specific site. e.g. if I have a static generated site, I could include zstd dictionary training as part of the build process to generate an optimal dictionary file one time, and then have the smallest download sizes possible, even less so than gzip.
To this end, there is a problem currently on a specification level (not to mention implementation level...) - there is no way for client and server to automatically coordinate on a dictionary file. The use of a dictionary is currently only possible if both ends know and set the dictionary in advance, which very much makes dictionaries unusable.
**Describe the solution you'd like**
I think the best way would be to register with IANA a `/.well-known/` location with a dictionary ID map... Let's say `/.well-known/zstd-dict-ids.json`. As the name suggests, it would be a JSON file. Why JSON and not a custom even more compact format? To enable easier auto generation via existing tooling.
It might f.e. have the following form:
```json
[
{
"p": "/blog",
"d": {
"": "/url/of/default/dictionary",
"32768": "/url/of/dictionary/for/id/32768",
"32769": "/url/of/dictionary/for/id/32769"
}
}
]
```
i.e. an array, each member being an object with an ID map (the key `d`) for specific path prefixes (the key `p`), with the path prefix defaulting to "/", i.e. "all files".
I'm not strongly attached to this exact JSON schema, but it being JSON and allowing for different URLs to use different dictionaries for the same dictionary ID is I think would be great. JSON as format for easier generation as already stated, and reusing IDs for different paths would be useful in environments where files pre-generated from different sources are being aggregated. I get that in a typical scenario, a user will have just one set for all of their files (just different IDs per file type). Anyway...
User agents would be expected to access that location for each origin on which they find a zstd response. Once they have the map, they can download the dictionary needed for the resource, and also pre-fetch other dictionaries in the background before requests for such files are even made. The map and each dictionary can be cached according to their respective caching headers, like any other resource.
If a user agent doesn't have a caching mechanism, or caching is otherwise hindered/disabled, it can still perform this procedure, though that does mean 3 HTTP requests (the compressed file, the map, the dictionary) per resource instead of just 1 or 2, but this is well worth it for user agents that do have caching, where they only make N (compressed files) + 1 (map) + M (dictionaries; typically much fewer than the resources, at worst equal to the number of resources) requests.
**Describe alternatives you've considered**
I thought about adding an option to the Transfer-Encoding HTTP header, along with the format, e.g.
```Transfer-Encoding: zstd;d=/url/to/dictionary```
but depending on the number of requests, size of each resource and length of the URL of the dictionary, the benefit of including the dictionary file path may outweigh the gains from a custom dictionary compared to a standardized one.
And that's not counting issues like the ones discussed in https://github.com/facebook/zstd/issues/2713 with modifying the HTTP protocol to support this.
I guess this extra header parameter is not mutually exclusive with a dictionary ID map... If both were supported and provided, I guess the header would just take precedence.
**Additional context**
The above are all just ideas on how to solve this, but I believe some sort of a solution to the custom dictionary problem will be needed before a wide adoption in browsers and non-browser HTTP user agents, as zstd's unique selling point compared to gzip/brotli is just hardly realized without dictionaries in place.
コントリビューションガイド
評価
この issue はまだ評価されていません。