web-platform-tests / web-platform-tests/interop

Add a way to exclude individual testharness tests

Open
#1,260 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

meta
Dominant language
JavaScript
Stars
524
Forks
35
Avg merge
1d 4h
Merged PRs (30d)
10

Description

The following two things are in conflict with each other:

  • An active focus area wanting to govern a test (file), freezing its scope.
  • A WG operating as normal, developing/extending a specification, necessarily increasing scope.

When some feature aspect is extended in a specification, there is often an existing feature-aspect.html testharness WPT where it makes the most sense to place a test (test()) for this extension. However, doing so would inadvertently increase the scope of the active focus area.

We currently have these choices:

  • Move the test for the new extension away from its natural habitat, i.e. a new file, artificially. (Undesirable)
  • Exclude the whole test from the active focus area, possibly allowing a minor extension to "poison" all of feature-aspect.html. (Undesirable)
  • Run the new test (test()) conditionally, based on feature detection (CSS.supports(), etc). (Undesirable, though arguably the least bad of these three.)

This might be a general issue that goes beyond testharness.js, but I think most of the pain would go away if we could somehow mark individual test()s as excluded. cc @foolip

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the testharness.js context and the issue's examples involving individual test() calls in existing feature-aspect.html files. Define how an individual test can be excluded from an active focus area without moving it or making it conditional. Done means the approach addresses the stated conflict while preserving normal specification development.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript
Domain
testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.