Avoiding timeouts in crash/reftests where a feature is unimplemented

Timing out is frequently one of the worst outcomes of a test—a large number of tests timing out inevitably wastes a lot of machine time, as the causes of time outs often occur quite quickly and thus the machine is sitting waiting for the timer.

We frequently see this in crash and reftests uses test-wait, with patterns such as:

unsupportedCallbackAPI(() => document.documentElement.className = '');

If unsupportedCallbackAPI is unsupported, then the test-wait class will never get removed, and the test will simply timeout.

What's unclear is whether we want to encourage people to write tests such as:

if ("unsupportedCallbackAPI" in window) {
  unsupportedCallbackAPI(() => document.documentElement.className = '');
} else {
  document.documentElement.className = '';
}

…and then trust the initial state of the page doesn't match the reference. But in crash tests this is effectively not running the actual test—though it indeed not crashing. Maybe it is okay to just skip the actual test code in such circumstances?

(Inspired by https://github.com/web-platform-tests/wpt/pull/41576 among others, for the record)

Start by reviewing the test-wait pattern in crash and reftests and the WPT pull request 41576 cited in the issue. Compare the timeout and unsupported-feature cases, then define an agreed RFC recommendation for how these tests should behave and what authors should write.