[Feature]: Conditional retries

🚀 Feature Request

Ability to provide a custom function that can decide whether the test should be retried or not, based on previous attempts.

Nice to have: ability to provide custom retry strategy as well.

Example

Usage example could be sth like this:

{
  retries: 3, // max number of retries
  retryStrategy: 'immediate',
  retryIf: (testInfo, errors) => {
    if (testInfo.retry <= 1) return true; // always retry at least once

    const errorsToRetry = [/Target crashed/, /browser has been closed/, /* ... */];
    for(const pattern of errorsToRetry) {
      if (errors.at(-1).match(pattern)) return [true, 'isolated']; // retry with strategy "isolated" if the error seems to be caused by env instability
    }
    return false; // don't retry more than once on other errors
  }
}

Motivation

In projects where lots of tests are being run periodically (my case: ~6000), sometimes it's just the remote cluster having hiccups and the entire run fails because of it. It disrupts devs' work and takes unnecessary time to keep checking those errors and retrying the entire ~2h pipeline because of an unexpected browser crash.

Why not just put retries: 5 or so? Because then it eats up unnecessary time and resources when there are genuine failures, especially on pipelines like ones checking PRs etc, that are expected to just have lots of genuinely failing tests.

Start by tracing Playwright's existing retry configuration and test-runner retry handling. Use the issue's retryIf and retryStrategy example to define the supported callback inputs, return values, and strategy behavior, then verify that conditional retries and custom strategies work without adding unnecessary retries for genuine failures.