web-platform-tests / web-platform-tests/wpt

Tests that compete for globally shared resources are flaky when run in parallel

Open
#44,972 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

infra priority:backlog
Dominant language
HTML
Stars
6.2k
Forks
3.9k
Avg merge
1d 6h
Merged PRs (30d)
586

Description

Originally filed as https://bugzilla.mozilla.org/show_bug.cgi?id=1866605

In that example we have two tests that use the clipboard. When run in parallel the tests both writing to and reading from the clipboard have an obvious race condition which causes intermittent failures. Our usual approach of partitioning the state doesn't work here because it's OS-level global state; partitioning would be equivalent to requiring tests to be run single-process-per-OS-instance.

To solve this the obvious abstract requirement is some way to annotate which tests are known to require the same shared state. For example:
<meta name=state content=clipboard>

Functionally it would then be up to the runner to ensure that multiple tests declaring the same state requirements don't run in parallel. This could be achieved by each type of state being associated with a mutex that must be acquired before starting the test (should be OK if the number of tests is small and contention is unlikely) or by scheduling such that those tests are never run in parallel.

Requirements wise a question is whether there's a case where we want vendor-specific global state annotations in expectation metadata files. This could be required if a certain browser has implementation-specific global state (e.g. a case where using the media stack from multiple processes at once causes issues).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No file or test entry point is named. Start by locating the test runner's expectation-metadata handling and parallel scheduling, then review how the clipboard example is represented. Done means reaching an agreed design for shared-state annotations and runner behavior, including vendor-specific metadata requirements.

Written by the indexing model from the issue text.

Assessment

Tech stack
html
Domain
testing-qa, tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.