web-platform-tests / web-platform-tests/interop

Scoring: variants and multi-globals

Open

#256 opened on Dec 20, 2022 7 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Description

One suggestion for webcodecs was to use all the tests which match the search video: https://wpt.fyi/results/webcodecs?label=master&label=experimental&aligned&view=subtest&q=video

However, due to extensive use of multi-global tests and variants this ends up not working particularly well. Most obviously, webcodecs/videoDecoder-codec-specific.https.any.js ends up contributing 20% to the overall score, as the one file contributing the following ten tests (out of a total of 48 tests):

This isn't the first time we've had problems like this with our scoring, but I think this is a much more extreme case than we've had otherwise.

Contributor guide

No contributing guide indexed for this repository

First steps

Read the whole issue, then the project's contributing guide.
Comment on the issue to say you are picking it up — it saves two people doing the same work.
Fork the repository and make your change on a branch.
Open a pull request that references the issue number.

Research direction

Start by reproducing the video search described in the issue and inspect the listed videoDecoder-codec-specific.https.any.js and .any.worker.html variants. Review the scoring entry point that aggregates these results; done means multi-global and variant tests no longer distort the overall score so severely.

Written by the indexing model from the issue text.

Assessment

Tech stack: javascript
Domain: testing-qa
Issue type: Feature
Difficulty: 5/5
Estimated time: Over a week
Activity status: Stale
Clarity: Needs clarification
Newbie friendliness: 30/100