bazelbuild / bazelbuild/bazel

flaky attribute and --flaky_test_attempts are disconnected; BUILD cannot express per-target retry counts

Open
#30,108 0 comments 1 reaction 0 assignees View on GitHub
team-Core type: bug untriaged
Dominant language
Java
Stars
25.8k
Forks
4.6k
Avg merge
2d 20h
Merged PRs (30d)
72

Description

## Problem

Bazel has two mechanisms for flaky-test retries that do not compose:

1. **`flaky` BUILD attribute** — boolean (`flaky = 0/1`). When `--flaky_test_attempts=default` (the built-in default), `flaky=1` gives a hardcoded **3** attempts; `flaky=0` gives **1**. The attribute cannot express degrees of flakiness (`flaky=2` is rejected at analysis: *expected one of [False, True, 0, 1]*).

2. **`--flaky_test_attempts`** — global or `regex_filter@N` retry count. A bare integer (e.g. `--flaky_test_attempts=3`) is parsed as catch-all `.*@3` and applies to **every** target, ignoring the `flaky` attribute entirely.

In practice CI almost always sets a global `--flaky_test_attempts=N`, which makes `flaky=1` on individual targets meaningless for execution. The only per-target alternative is long `regex_filter@N` lists, which do not scale when flaky targets are scattered across a large monorepo (not grouped by package).

Meanwhile `--runs_per_test` with per-target regex can express per-target parallelism, but has the same regex-scaling problem and different semantics (flake detection, not failure-triggered retries).

## Expected behavior

Per-target retry policy should be declarable in BUILD files (or tags), composable with a global CI baseline, e.g.:

- Non-flaky targets: 1 attempt regardless of global flag.
- Flaky targets: `attempts = baseline_from_flag + extra_from_build_attribute` (or similar), without maintaining package-level regex lists.

## Current code references

- `TestStrategy.getTestAttempts()` — branches on `isFlaky()` then calls `getTestAttemptsPerLabel()` with default 1 or 3.
- `ExecutionOptions.TestAttemptsConverter.parseAsInteger()` — bare integer becomes `.*` catch-all, overriding attribute.
- `Type.BooleanType.convert()` — rejects `flaky=2`.
- Docs (`site/en/docs/user-manual.md`): *"only a single attempt for regular tests, and 3 for test rules with the flaky attribute set"* — only true when flag is `default`.

## Related

- #3783 (closed) — confusion about `--flaky_test_attempts` semantics
- #29897 (open) — per-target `--test_timeout` via regex (same scaling concern)
- PR #28635 (open) — raises max attempts cap, unrelated to per-target policy

## Proposal (sketch)

1. Change `flaky` from `boolean` to `int` (0 = not flaky; 1 = legacy true; N>1 = additional attempts or total tier).
2. When `--flaky_test_attempts=K` is set globally, non-flaky targets stay at 1 attempt; flaky targets use `K + flaky` (exact semantics TBD).
3. Optionally: parallel hedged retries (separate FR).

Happy to contribute a PR if there is agreement on semantics and backward compatibility.

Contributor guide

Open the contributing guide

Research direction

Start with TestStrategy.getTestAttempts(), ExecutionOptions.TestAttemptsConverter.parseAsInteger(), and Type.BooleanType.convert(), then compare the user-manual.md wording with their current behavior. Clarify the retry arithmetic, boolean compatibility, and interaction between global and per-target settings before implementation; done means the agreed policy works without regex lists and is documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
build-system, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.