web-platform-tests / web-platform-tests/wpt
[wptrunner] Inconsistent behavior for "jsshell" tests
Nobody has claimed this yet.
- Dominant language
- HTML
- Stars
- 6.2k
- Forks
- 3.9k
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 586
Description
Earlier this year, a feature was added to the WPT CLI to support executing a new kind of test. Yesterday, the first of these tests were introduced into WPT. The addition of these tests uncovered an inconsistent behavior.
The "jsshell" version of these tests cannot be executed by any code available in WPT itself, so the WPT CLI ignores them during normal test execution. This means it does not attempt to run them, and it also does not include them in its various output logs. However, other features interact with the tests (e.g. --list-tests, --list-disabled, --list-test-groups, and --verify), but these do not ignore the "jsshell" versions.
This inconsistency may cause confusion for human operators, and it interferes with scripted applications of the WPT CLI (see https://github.com/web-platform-tests/results-collection/issues/590).
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing how the WPT CLI handles "jsshell" tests during normal execution, then compare that behavior with --list-tests, --list-disabled, --list-test-groups, and --verify. Done means these features consistently include or ignore the tests according to the intended execution behavior, without confusing operators or scripted consumers.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript
- Domain
- cli, testing-qa
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100