web-platform-tests / web-platform-tests/wpt
Threshold in user-timing tests too short for slow machines
Nobody has claimed this yet.
- Dominant language
- HTML
- Stars
- 6.2k
- Forks
- 3.9k
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 586
Description
When we (Chrome/Blink) run the user-timing tests as part of our continuous integration, we frequently test failures on the user-timing tests due to steps exceeding the threshold, e.g.:
window.performance.getEntriesByName("measure_nav_start_no_end")[0].duration ~== 282 (up to 20ms difference allowed) assert_true: window.performance.getEntriesByName("measure_nav_start_no_end")[0].duration ~== 282 (up to 20ms difference allowed) expected true got false
window.performance.getEntriesByName("measure_nav_start_no_end")[0].duration ~== 225 (up to 20ms difference allowed) assert_true: window.performance.getEntriesByName("measure_nav_start_no_end")[0].duration ~== 225 (up to 20ms difference allowed) expected true got false
etc.
This occurs randomly, but most frequently on slow machines: older Windows (e.g. XP), older OSX (e.g. 10.6, 10.7) with debug builds, etc.
In the short term we'll probably just stop running these tests, but it would be nice to have an upstream solution. Simply upping the 20ms threshold would reduce the number of failures, but any suggestions welcome.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Locate the user-timing tests and the assertions comparing performance.getEntriesByName("measure_nav_start_no_end")[0].duration with expected values. Run them on a slow or debug environment to reproduce the intermittent failures, then define a stable threshold strategy and verify that the tests no longer fail spuriously without weakening their coverage.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- html, javascript
- Domain
- testing-qa, web-dev
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100