Change the order of preURL and preScript when running the tests

Hello,
Could you please consider changing the order of executing preURL and preScript?

Motivation

I'm currently measuring performance of cached pages and I spotted the problem when using only preURL parameter: visual metrics (FirstVisualChange, SpeedIndex etc.) always show 0ms. That's caused by the way Chrome works (it does not "reset" the page layout between preUrl and the actual page to test). Visual metrics that base mainly on the recording are just not working (please see reproduction steps below). This is also clearly visible on the video recording that is visible on the report.

I have worked out a solution with using preScript (Approach#2), but it could be much easier if the script only clears the layout and I can still use preUrl to load the cache. Currently it is impossible, because preScript is executed before preUrl (see Approach#3).

Environment details

• Running with Docker
• MacOS Catalina (10.15.5)
• Sitespeed.io: 13.3.2 (latest at the moment)

For your convenience I have created repo with all the code I present below:

• https://github.com/Skejven/test-page-with-cache

Approach 1 - only preUrl

Visual metrics won't work as expected (see explanation above).

docker run --shm-size=2g --rm -v "$(pwd)":/sitespeed.io sitespeedio/sitespeed.io:13.3.2  \
   --preURL https://www.sitespeed.io/support/?warmup=true \
    --urlAlias Homepage https://www.sitespeed.io/support/

Results:

Google Chrome 83.0.4103.61
Mozilla Firefox 77.0
[2020-07-01 12:28:37] INFO: Versions OS: linux 4.9.125-linuxkit nodejs: v12.16.2 sitespeed.io: 13.3.2 browsertime: 8.14.0 coach: 5.0.1
[2020-07-01 12:28:37] INFO: Running tests using Chrome - 3 iteration(s)
[2020-07-01 12:28:38] INFO: Accessing preURL https://www.sitespeed.io/support/?warmup=true
[2020-07-01 12:28:46] INFO: Testing url https://www.sitespeed.io/support/ iteration 1
[2020-07-01 12:29:05] INFO: https://www.sitespeed.io/support/ TTFB: 69ms DOMContentLoaded: 109ms firstPaint: 121ms FCP: 121ms LCP: 121ms Load: 112ms
[2020-07-01 12:29:05] INFO: VisualMetrics: FirstVisualChange: 0ms SpeedIndex: 0ms PerceptualSpeedIndex: 0ms ContentfulSpeedIndex: 0ms VisualComplete85: 0ms LastVisualChange: 0ms
[2020-07-01 12:29:06] INFO: Accessing preURL https://www.sitespeed.io/support/?warmup=true
[2020-07-01 12:29:14] INFO: Testing url https://www.sitespeed.io/support/ iteration 2
[2020-07-01 12:29:25] INFO: https://www.sitespeed.io/support/ TTFB: 37ms DOMContentLoaded: 74ms firstPaint: 82ms FCP: 82ms LCP: 82ms Load: 77ms
[2020-07-01 12:29:25] INFO: VisualMetrics: FirstVisualChange: 0ms SpeedIndex: 0ms PerceptualSpeedIndex: 0ms ContentfulSpeedIndex: 0ms VisualComplete85: 0ms LastVisualChange: 0ms
[2020-07-01 12:29:26] INFO: Accessing preURL https://www.sitespeed.io/support/?warmup=true
[2020-07-01 12:29:34] INFO: Testing url https://www.sitespeed.io/support/ iteration 3
[2020-07-01 12:29:45] INFO: https://www.sitespeed.io/support/ TTFB: 31ms DOMContentLoaded: 63ms firstPaint: 72ms FCP: 72ms LCP: 72ms Load: 66ms
[2020-07-01 12:29:45] INFO: VisualMetrics: FirstVisualChange: 0ms SpeedIndex: 0ms PerceptualSpeedIndex: 0ms ContentfulSpeedIndex: 0ms VisualComplete85: 0ms LastVisualChange: 0ms
[2020-07-01 12:29:45] INFO: https://www.sitespeed.io/support/ 1 requests, TTFB: 46ms (±9.63ms), firstPaint: 92ms (±12.21ms), firstVisualChange: 0ms (±0.00ms), FCP: 92ms (±12.21ms), DOMContentLoaded: 82ms (±11.32ms), LCP: 92ms (±12.21ms), CLS: 0 (±0.00), Load: 85ms (±11.32ms), speedIndex: 0ms (±0.00ms), perceptualSpeedIndex: 0ms (±0.00ms), contentfulSpeedIndex: 0ms (±0.00ms), visualComplete85: 0ms (±0.00ms), lastVisualChange: 0ms (±0.00ms) (3 runs)
[2020-07-01 12:29:48] INFO: HTML stored in /sitespeed.io/sitespeed-result/www.sitespeed.io/2020-07-01-12-28-37

Approach 2 - only preScript

This is what I'm currently using:

docker run --shm-size=2g --rm -v "$(pwd)":/sitespeed.io sitespeedio/sitespeed.io:13.3.2 \
    --preScript warm_up_cache.js --browsertime.urlToWarmUp https://www.sitespeed.io/support/ \
    --urlAlias Homepage https://www.sitespeed.io/support/

warm_up_cache.js

module.exports = async function(context, commands) {
    let urlToWarmUp = context.options.urlToWarmUp;
    context.log.info("Warming up cache for page: " + urlToWarmUp);
    await commands.navigate(urlToWarmUp + "?warmup=true");
    // Remove body of the page for visual metrics to start with blank view
    await commands.js.run('document.body.innerHTML = ""');
};

Results:

Google Chrome 83.0.4103.61
Mozilla Firefox 77.0
[2020-07-01 12:29:51] INFO: Versions OS: linux 4.9.125-linuxkit nodejs: v12.16.2 sitespeed.io: 13.3.2 browsertime: 8.14.0 coach: 5.0.1
[2020-07-01 12:29:52] INFO: Running tests using Chrome - 3 iteration(s)
[2020-07-01 12:29:52] INFO: Warming up cache for page: https://www.sitespeed.io/support/
[2020-07-01 12:29:52] INFO: Navigating to url https://www.sitespeed.io/support/?warmup=true iteration 1
[2020-07-01 12:29:59] INFO: Testing url https://www.sitespeed.io/support/ iteration 1
[2020-07-01 12:30:23] INFO: https://www.sitespeed.io/support/ TTFB: 58ms DOMContentLoaded: 98ms firstPaint: 111ms FCP: 111ms LCP: 111ms Load: 103ms
[2020-07-01 12:30:23] INFO: VisualMetrics: FirstVisualChange: 135ms SpeedIndex: 135ms PerceptualSpeedIndex: 135ms ContentfulSpeedIndex: 135ms VisualComplete85: 135ms LastVisualChange: 135ms
[2020-07-01 12:30:24] INFO: Warming up cache for page: https://www.sitespeed.io/support/
[2020-07-01 12:30:24] INFO: Navigating to url https://www.sitespeed.io/support/?warmup=true iteration 2
[2020-07-01 12:30:31] INFO: Testing url https://www.sitespeed.io/support/ iteration 2
[2020-07-01 12:30:43] INFO: https://www.sitespeed.io/support/ TTFB: 32ms DOMContentLoaded: 88ms firstPaint: 100ms FCP: 100ms LCP: 100ms Load: 92ms
[2020-07-01 12:30:43] INFO: VisualMetrics: FirstVisualChange: 125ms SpeedIndex: 125ms PerceptualSpeedIndex: 125ms ContentfulSpeedIndex: 125ms VisualComplete85: 125ms LastVisualChange: 125ms
[2020-07-01 12:30:44] INFO: Warming up cache for page: https://www.sitespeed.io/support/
[2020-07-01 12:30:44] INFO: Navigating to url https://www.sitespeed.io/support/?warmup=true iteration 3
[2020-07-01 12:30:50] INFO: Testing url https://www.sitespeed.io/support/ iteration 3
[2020-07-01 12:31:03] INFO: https://www.sitespeed.io/support/ TTFB: 34ms DOMContentLoaded: 69ms firstPaint: 84ms FCP: 84ms LCP: 84ms Load: 72ms
[2020-07-01 12:31:03] INFO: VisualMetrics: FirstVisualChange: 100ms SpeedIndex: 100ms PerceptualSpeedIndex: 100ms ContentfulSpeedIndex: 100ms VisualComplete85: 100ms LastVisualChange: 100ms
[2020-07-01 12:31:03] INFO: https://www.sitespeed.io/support/ 1 requests, TTFB: 41ms (±6.82ms), firstPaint: 98ms (±6.40ms), firstVisualChange: 120ms (±8.50ms), FCP: 98ms (±6.40ms), DOMContentLoaded: 85ms (±6.94ms), LCP: 98ms (±6.40ms), CLS: 0 (±0.00), Load: 89ms (±7.41ms), speedIndex: 120ms (±8.50ms), perceptualSpeedIndex: 120ms (±8.50ms), contentfulSpeedIndex: 120ms (±8.50ms), visualComplete85: 120ms (±8.50ms), lastVisualChange: 120ms (±8.50ms) (3 runs)
[2020-07-01 12:31:06] INFO: HTML stored in /sitespeed.io/sitespeed-result/www.sitespeed.io/2020-07-01-12-29-51

Approach 3 - use both preUrl and preScript

This is what I would like to have (won't work at the moment because of the order of executing preUrl and preScript):

docker run --shm-size=2g --rm -v "$(pwd)":/sitespeed.io sitespeedio/sitespeed.io:13.3.2 \
    --preURL https://www.sitespeed.io/support/?warmup=true \
    --preScript clear_view.js \
    --urlAlias Homepage https://www.sitespeed.io/support/

clear_view.js

module.exports = async function(context, commands) {
    // Remove body of the page for visual metrics to start with blank view
    context.log.info("-----------------> Clearing view <----------------");
    await commands.js.run('document.body.innerHTML = ""');
};

Results:

Google Chrome 83.0.4103.61
Mozilla Firefox 77.0
[2020-07-01 12:31:09] INFO: Versions OS: linux 4.9.125-linuxkit nodejs: v12.16.2 sitespeed.io: 13.3.2 browsertime: 8.14.0 coach: 5.0.1
[2020-07-01 12:31:10] INFO: Running tests using Chrome - 3 iteration(s)
[2020-07-01 12:31:10] INFO: -----------------> Clearing view <----------------
[2020-07-01 12:31:10] INFO: Accessing preURL https://www.sitespeed.io/support/?warmup=true
[2020-07-01 12:31:18] INFO: Testing url https://www.sitespeed.io/support/ iteration 1
[2020-07-01 12:31:31] INFO: https://www.sitespeed.io/support/ TTFB: 34ms DOMContentLoaded: 72ms firstPaint: 83ms FCP: 83ms LCP: 83ms Load: 74ms
[2020-07-01 12:31:31] INFO: VisualMetrics: FirstVisualChange: 0ms SpeedIndex: 0ms PerceptualSpeedIndex: 0ms ContentfulSpeedIndex: 0ms VisualComplete85: 0ms LastVisualChange: 0ms
[2020-07-01 12:31:31] INFO: -----------------> Clearing view <----------------
[2020-07-01 12:31:31] INFO: Accessing preURL https://www.sitespeed.io/support/?warmup=true
[2020-07-01 12:31:39] INFO: Testing url https://www.sitespeed.io/support/ iteration 2
[2020-07-01 12:31:51] INFO: https://www.sitespeed.io/support/ TTFB: 36ms DOMContentLoaded: 87ms firstPaint: 93ms FCP: 93ms LCP: 93ms Load: 89ms
[2020-07-01 12:31:51] INFO: VisualMetrics: FirstVisualChange: 0ms SpeedIndex: 0ms PerceptualSpeedIndex: 0ms ContentfulSpeedIndex: 0ms VisualComplete85: 0ms LastVisualChange: 0ms
[2020-07-01 12:31:51] INFO: -----------------> Clearing view <----------------
[2020-07-01 12:31:51] INFO: Accessing preURL https://www.sitespeed.io/support/?warmup=true
[2020-07-01 12:31:59] INFO: Testing url https://www.sitespeed.io/support/ iteration 3
[2020-07-01 12:32:11] INFO: https://www.sitespeed.io/support/ TTFB: 33ms DOMContentLoaded: 121ms firstPaint: 112ms FCP: 112ms LCP: 112ms Load: 124ms
[2020-07-01 12:32:11] INFO: VisualMetrics: FirstVisualChange: 0ms SpeedIndex: 0ms PerceptualSpeedIndex: 0ms ContentfulSpeedIndex: 0ms VisualComplete85: 0ms LastVisualChange: 0ms
[2020-07-01 12:32:11] INFO: https://www.sitespeed.io/support/ 1 requests, TTFB: 34ms (±0.72ms), firstPaint: 96ms (±6.94ms), firstVisualChange: 0ms (±0.00ms), FCP: 96ms (±6.94ms), DOMContentLoaded: 93ms (±11.84ms), LCP: 96ms (±6.94ms), CLS: 0 (±0.00), Load: 96ms (±12.10ms), speedIndex: 0ms (±0.00ms), perceptualSpeedIndex: 0ms (±0.00ms), contentfulSpeedIndex: 0ms (±0.00ms), visualComplete85: 0ms (±0.00ms), lastVisualChange: 0ms (±0.00ms) (3 runs)
[2020-07-01 12:32:14] INFO: HTML stored in /sitespeed.io/sitespeed-result/www.sitespeed.io/2020-07-01-12-31-09

Trace the test-runner handling for the --preURL and --preScript options, using the command examples and clear_view.js behavior in the issue as the reproduction. Verify the execution order and visual-metrics output with both options enabled; done means preURL runs before preScript and the resulting metrics are no longer zero.