WebAssembly / WebAssembly/spec

Simplify JS-API test syncing

Open
#2,130 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
WebAssembly
Stars
3.5k
Forks
539
Avg merge
2d 1h
Merged PRs (30d)
13

Description

The JS-API testing situation is complicated. We've got diverging copies in spec/proposals and WPT. I'd like to simplify this to make contributing new tests easier.

Here's my understanding:

  1. We have JS-API tests in test/js-api of this repo. These are just WPTs and use the same harness setup.
  2. Proposals add their own JS-API tests to their forks of this repo.
  3. The web-platform-test repo has copies of the tests from the spec and proposal repos in tests/wasm/jsapi.
  4. Right now there is no automation between any of these repos, and so of course they've diverged over time.
    a. Spec contains about 7 tests that have never been pushed to WPT
    b. WPT has 'tentative' tests from the various proposals that are not yet in spec (this is expected)
    c. WPT has 4 non-tentative tests that have never been pushed to spec (this is not expected)
    d. Older proposals are also stale and have diverging harnesses and other infrastructure (like wasm module builder).

The core issue is that there's no canonical repo to prevent divergence over time.

Here are the two options I can think of:

1. Make 'spec' repo canonical

New JS-API tests must land in 'spec' first. We then automate a sync from 'spec' to 'wpt'.

This solves the divergence problem for 'spec' but not for the proposal repos.

Maybe they continue to be manually pushed to WPT? And then when a proposal gets merged, we manually remove them from WPT and replace them with automatically synced version.

2. Make 'wpt' repo canonical.

New JS-API tests must land in 'WPT' first, including tentative ones.

We then automate a sync from 'wpt' to 'spec'. The sync process needs to filter out 'tentative' tests from other proposals.

We could also have an automated sync from 'wpt' to individual proposals that preserves the 'tentative' tests from their proposal (maybe use subfolders for this).

2.a And only have JS-API tests in 'wpt'

Go even further than option 2, and just drop the JS-API copies from spec/proposals. Only use the WPT repo for tests.

Firefox only uses the WPT repo. We never directly use the JS-API tests from the spec/proposal repos. Maybe other browsers are different?

This would let us get rid of any complicated syncing, and just have one canonical location.

I have a preference for option 2.a, but I'm interested in what others think.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by comparing the JS-API tests in test/js-api with the copies in WPT's tests/wasm/jsapi, including the proposal and harness differences described in the issue. Review the three proposed repository arrangements and determine which location should be canonical and how tentative tests are handled. Done means a decided synchronization plan with clear repository ownership and automation boundaries.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, wasm
Domain
testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.