Benchmark compare last release: asv silently runs zero benchmarks since #10866 (py-rattler was added to environment-benchmark.yml but not environment.yml)

What is your issue?

Every push-to-main run of Benchmark compare last release concludes success in ~30 seconds, but asv exits before selecting an environment and no benchmark is ever executed — the compare-against-last-release performance alarm is silently dead. From today's green run (32095799355, 30 s):

· Unknown environment type 'rattler'. Allowed values based on existing plugins are ['existing', 'conda', 'uv', 'virtualenv'].
· No environments selected

Cause. #10866 (2025-10-22, 19f2973) switched asv_bench/asv.conf.json to "environment_type": "rattler" and added py-rattler to ci/requirements/environment-benchmark.yml#L15 — but benchmarks-last-release.yml builds its environment from ci/requirements/environment.yml, which never got py-rattler, so the asv it installs has no rattler plugin. The job can't see asv's non-zero exit: the run step pipes asv continuous … | sed … | tee benchmarks.log without pipefail, and the only failure gate is the grep at benchmarks-last-release.yml#L65-L70 — "Traceback \|failed\|PERFORMANCE DECREASED" — which an environment-construction failure matches never, so the step exits 0 and the run stays green.

Run-history evidence (public Actions history, verified 2026-08-18): all 450 retained runs (2025-07-02 → today) completed in ≤ ~4 minutes — none long enough to have executed the suite (a real asv continuous in the sibling Benchmark workflow takes 60–90 min, e.g. 31615795501 on #11239). 448 concluded success; the only two non-green runs (21605156467, 21606708075, 2026-02-02) were cancelled during environment setup and never reached asv. Log spot-checks across the 90-day retention window all show the identical rattler error under a green run: 26782017237 (06-01), 28122316168 (06-24), 32095799355 (08-18). Older logs have expired, so I can't state the failure mode before #10866 — but the durations show no retained run ever executed benchmarks on either side of it.

The label-triggered Benchmark workflow is not affected: its env file got py-rattler in #10866 and it genuinely executes the suite.

Verification on a fork (main synced with c2998a75): two workflow_dispatch runs of the unmodified workflow reproduce the green no-op with the same log lines: 32199849622, 32199928875.

Proposed fix — add py-rattler to ci/requirements/environment.yml, the same one-line change #10866 made to environment-benchmark.yml. Separately, set -o pipefail on the run step (or checking asv's exit status directly) would have turned this red in October instead of green for ten months. Happy to open a PR for either or both if preferred.

AI disclosure: this issue was researched and written by an AI agent, explicitly instructed to investigate and file it; all run IDs, durations, log lines, and diffs quoted above were verified against the public GitHub API before filing, and the fork runs were executed as part of that verification. [This is Claude Code on behalf of @glaziermag]

Start with .github/workflows/benchmarks-last-release.yml and compare its environment setup with ci/requirements/environment-benchmark.yml and ci/requirements/environment.yml. Reproduce the workflow or inspect the cited run logs, then verify that the last-release job selects a rattler environment, executes benchmarks, and reports an asv failure instead of staying green.