web-platform-tests / web-platform-tests/interop
Document the data collection and results generation system
Nobody has claimed this yet.
- Dominant language
- JavaScript
- Stars
- 524
- Forks
- 35
- Avg merge
- 1d 4h
- Merged PRs (30d)
- 10
Description
The way that we go from tests to results to data on wpt.fyi to interop scores is under documented an unclear. There are two specific problems here (which maybe require different solutions, but it's worth considering as part of one higher level issue):
- People don't know what to expect e.g. in terms of latency, and don't know at what point to raise an alarm when something isn't working the way they expect (e.g. when browser changes aren't showing up on the interop dashboard).
- It's unclear what parts of the pipeline are considered stable and can be relied on by external tooling (e.g. for vendors wanting to understand how specific in-progress changes affect interop).
That suggests that for 2024 we should consider two improvements:
- Produce documentation of the overall dataflow, so that there's a reference to point people to for all the systems involved, what they should expect in terms of latency, and what to do when something doesn't work in the way they expect.
- Provide a stable API for accessing the data and tooling needed to recreate the interop scores. Currently there are a variety of data files in different, undocumented, formats, that one must download from different systems, which are quite specialised to the needs of wpt.fyi. That makes external tooling both difficult to create in the first place, and liable to break whenever the needs of wpt.fyi change.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by mapping the systems and data files involved in moving from tests to results, wpt.fyi data, and Interop scores. Document expected latency, failure investigation steps, and which pipeline parts are stable for external tooling. Define what a stable data API and score-recreation workflow must provide.
Written by the indexing model from the issue text.
Assessment
- Domain
- api, data-engineering, documentation
- Issue type
- Documentation
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100