web-platform-tests / web-platform-tests/rfcs
Should tentative tests require links?
Nobody has claimed this yet.
- Dominant language
- No language data
- Stars
- 110
- Forks
- 90
- PR merge metrics
- No merged PRs in 30d
Description
For background, see my prior comment on https://github.com/web-platform-tests/rfcs/pull/215#discussion_r1876443323:
[We have tentative tests where:]
- where there are WG resolutions not yet applied to the spec making tests tentative (e.g.,
/css/css-overflow/line-clamp/line-clamp-005.tentative.htmlis an example of this, with [css-overflow] Is continue: discard working in the fragment tree useful? w3c/csswg-drafts#7708 (comment)),- where there is a proposal to change the spec to some other behaviour (e.g.,
svg-aam/role/role-img.tentative.html, with SVG image role should align with HTML on missing versus empty alt w3c/svg-aam#32),- where there is a proposed addition to the spec (e.g.,
/html/dom/elements/global-attributes/the-anchor-attribute-001.tentative.html, with Add anchor attribute whatwg/html#9144),- where the spec is unclear (e.g,
webdriver/tests/bidi/input/release_actions/sequence_tentative.py, with Should a navigation clean-up thebrowsing context input state map? w3c/webdriver#1859).
It should be apparent in context why the test is tentative and what needs to be resolved to make it non-tentative.
I'd hope that tentative tests would include a link to a spec bug that needs to be resolved for the test to be non-tentative.
However:
% rg --glob '*tentative*' 'https?://((bugs\.webkit\.org|bugs\.chromium\.org|crbug\.com|issues\.chromium\.org)/|github\.com/.*/(issue|pull))' --files-without-match | wc -l
2078
…shows we have a very large number of tentative tests which don't link anywhere (even a vendor bug tracker!).
This makes it hard to audit the status of tentative tests, and what should and should not still be tentative.
An obvious option here is to require tentative tests to have a link — though what to is a reasonable question — which might be worth considering?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the tentative-test filename guidance in the linked documentation and inspect the reported rg search covering tests without issue or pull-request links. The issue does not identify an implementation file or settle what link types should be required, so completion would first require deciding the policy and its enforcement scope.
Written by the indexing model from the issue text.
Assessment
- Domain
- testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100