web-platform-tests / web-platform-tests/rfcs

Remove tests that rely on undefined testing APIs

Open
#172 7 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
No language data
Stars
110
Forks
90
PR merge metrics
No merged PRs in 30d

Description

WPT contains a variety of tests that rely on MojoJS to run in Chromium-based browsers.

Chromium's own WPT docs say:

Some specs may define testing APIs (e.g. WebUSB), which may be polyfilled with internal API like MojoJS. MojoJS is only allowed in WPT for this purpose. Please reach out to blink-dev@chromium.org before following the process below for adding a new test-only API: [.…]

However, we have many cases where things are "polyfilled" via MojoJS where the spec doesn't define the testing API, and nor has there been any attempt within WPT to document what the testing API the tests expect is. (Furthermore, https://groups.google.com/a/chromium.org/g/blink-dev/search?q=mojojs makes it look like basically nobody has ever followed those steps, as nobody has done the "reach out" step.)

The tests are, thus, practically Chromium-specific, short of someone reverse-engineering what the testing API is.

For example, the Shape Detection tests that were added to WPT in https://github.com/web-platform-tests/wpt/commit/dfc0080d132ac25c4cf824a25735203b3a2b6853 rely on /shape-detection/resources/shapedetection-helpers.js to load testing APIs, except nowhere in https://wicg.github.io/shape-detection-api/ (or any other document) are FaceDetectionTest, BarcodeDetectionTest, or TextDetectionTest actually defined.

Notwithstanding questions of whether any such testing API would ship in the browser, this means wpt.fyi will forever show Safari as failing many of the shape detection tests, even if the feature flag were enabled by default, unless we invest in reverse-engineering Chromium's testing API. Part of the very goal of standards and WPT is to avoid vendors having to invest in reverse-engineering one-another.

In extreme, /serial/README.md even notes:

there is no separate specification of the API other than the tests themselves and the Chromium implementation

WPT cannot be a dumping ground for inherently vendor-specific tests, written with little-to-no regard for other vendors using them, as this undermines the very premise of it being a shared test suite. It is unreasonable and unhelpful for Chromium engineers to export tests which other vendors must reverse-engineer the testing API to gain any benefit from. Chromium can maintain these tests in their own, non-exported tree, and do so until such point as they define what the testing APIs actually are.

As such, we should remove all tests that rely on undefined testing APIs from WPT.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by auditing WPT tests that load MojoJS, including shape-detection/resources/shapedetection-helpers.js, and read serial/README.md for the stated testing-API problem. Identify tests whose required APIs are neither specified nor documented, then remove those tests from WPT; completion means no such vendor-specific tests remain.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript
Domain
testing-qa
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.