web-platform-tests / web-platform-tests/interop

Document the data collection and results generation system

Open
#357 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

meta user-interface
Dominant language
JavaScript
Stars
524
Forks
35
Avg merge
1d 4h
Merged PRs (30d)
10

Description

The way that we go from tests to results to data on wpt.fyi to interop scores is under documented an unclear. There are two specific problems here (which maybe require different solutions, but it's worth considering as part of one higher level issue):

  • People don't know what to expect e.g. in terms of latency, and don't know at what point to raise an alarm when something isn't working the way they expect (e.g. when browser changes aren't showing up on the interop dashboard).
  • It's unclear what parts of the pipeline are considered stable and can be relied on by external tooling (e.g. for vendors wanting to understand how specific in-progress changes affect interop).

That suggests that for 2024 we should consider two improvements:

  • Produce documentation of the overall dataflow, so that there's a reference to point people to for all the systems involved, what they should expect in terms of latency, and what to do when something doesn't work in the way they expect.
  • Provide a stable API for accessing the data and tooling needed to recreate the interop scores. Currently there are a variety of data files in different, undocumented, formats, that one must download from different systems, which are quite specialised to the needs of wpt.fyi. That makes external tooling both difficult to create in the first place, and liable to break whenever the needs of wpt.fyi change.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by mapping the systems and data files involved in moving from tests to results, wpt.fyi data, and Interop scores. Document expected latency, failure investigation steps, and which pipeline parts are stable for external tooling. Define what a stable data API and score-recreation workflow must provide.

Written by the indexing model from the issue text.

Assessment

Domain
api, data-engineering, documentation
Issue type
Documentation
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.