web-platform-tests / web-platform-tests/wpt

Threshold in user-timing tests too short for slow machines

Open
#2,115 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

user-timing
Dominant language
HTML
Stars
6.2k
Forks
3.9k
Avg merge
1d 6h
Merged PRs (30d)
586

Description

When we (Chrome/Blink) run the user-timing tests as part of our continuous integration, we frequently test failures on the user-timing tests due to steps exceeding the threshold, e.g.:

window.performance.getEntriesByName("measure_nav_start_no_end")[0].duration ~== 282 (up to 20ms difference allowed) assert_true: window.performance.getEntriesByName("measure_nav_start_no_end")[0].duration ~== 282 (up to 20ms difference allowed) expected true got false

window.performance.getEntriesByName("measure_nav_start_no_end")[0].duration ~== 225 (up to 20ms difference allowed) assert_true: window.performance.getEntriesByName("measure_nav_start_no_end")[0].duration ~== 225 (up to 20ms difference allowed) expected true got false

etc.

This occurs randomly, but most frequently on slow machines: older Windows (e.g. XP), older OSX (e.g. 10.6, 10.7) with debug builds, etc.

In the short term we'll probably just stop running these tests, but it would be nice to have an upstream solution. Simply upping the 20ms threshold would reduce the number of failures, but any suggestions welcome.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Locate the user-timing tests and the assertions comparing performance.getEntriesByName("measure_nav_start_no_end")[0].duration with expected values. Run them on a slow or debug environment to reproduce the intermittent failures, then define a stable threshold strategy and verify that the tests no longer fail spuriously without weakening their coverage.

Written by the indexing model from the issue text.

Assessment

Tech stack
html, javascript
Domain
testing-qa, web-dev
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.