web-platform-tests / web-platform-tests/wpt

MANIFEST.json.zst files are bigger than bz2 files

Open
#33,489 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

infra priority:backlog
Dominant language
HTML
Stars
6.2k
Forks
3.9k
Avg merge
1d 6h
Merged PRs (30d)
586

Description

While looking into https://github.com/web-platform-tests/wpt/pull/33452 and trying to avoid the zstd binary as a dependency, I discovered that the MANIFEST.json.zst files are actually bigger than the bzip2 files we're uploading. In https://github.com/web-platform-tests/wpt/releases/tag/merge_pr_33467, we have:

MANIFEST.json.bz2: 2.08 MB (2180027 bytes)
MANIFEST.json.gz: 2.54 MB (2662852 bytes)
MANIFEST.json.zst: 2.09 MB (2190299 bytes)

In this case, zst is ~0.5% bigger than bz2.

bz2 and zst formats were introduced in https://github.com/web-platform-tests/wpt/pull/16083, but at the time zst was smaller:

MANIFEST.json.bz2: 1.94 MB
MANIFEST.json.gz: 2.37 MB
MANIFEST.json.zst: 1.86 MB

Since it requires an additional Python dependency we should reconsider if using zst is still really useful.

cc @gsnedders @jgraham

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No file or test is named. Start by locating the release path that produces and uploads MANIFEST.json.bz2, MANIFEST.json.gz, and MANIFEST.json.zst, then compare its zstd dependency and compression choices. Done means a decided compression approach with the release artifacts and dependency requirements aligned.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
release, tooling
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.