rstudio / rstudio/rstudio

E2E runs that select zero tests are reported as successful

Open
#18,428 1 comment 0 reactions 1 assignee Claimed by @ronblum View on GitHub
automation bug
Dominant language
Java
Stars
5.1k
Forks
1.2k
PR merge metrics
PR metrics pending

Description

### System details

RStudio Edition : N/A -- affects the Desktop and Server E2E engines alike
RStudio Version : N/A
OS Version : N/A -- affects every per-OS engine
R Version : N/A

### Steps to reproduce the problem

In any per-OS E2E engine workflow -- for example `os-test-e2e-rstudio-desktop-os-macos-14-arm64.yml`:

1. Go to Actions and select the engine workflow.
2. Click "Run workflow".
3. Enter `tests/smoke/startup.test.ts` in the `test_selector` field.
4. Click the green "Run workflow" button.
5. Wait for the run to finish.
- The run finishes with a green check and a `success` conclusion.
6. Open the `e2e-merge` job and expand its summary step.
- The log reads `Overall: passed=0 failed=0 skipped=0 flaky=0 rate=0%`.
- The sticky comment and job summary report the run as a pass.

### Describe the problem in detail

`e2e/rstudio/scripts/summarize-merged-report.mjs` totals the results, computes a pass rate, writes them as step outputs, logs them, and ends. It has no zero-test check and no non-zero exit, so a run where Playwright selected nothing looks identical to a run where everything passed. Playwright exits 0 when no tests match a selector, so nothing upstream catches it either.

This reaches every per-OS engine, since they all merge through the same summarizer.

The `test_selector` input makes it easy to hit by accident. `tests/smoke/startup.test.ts` is tagged `@smoke`, and `playwright.config.ts:80` grep-inverts `@smoke` unless `PW_RUN_SMOKE` is set -- which no engine workflow passes -- so that selector selects nothing at all.

This has already caused a real false pass. Run 30363343377 finished with a `success` conclusion while reporting `passed=0 failed=0 skipped=0 flaky=0 rate=0%`, and was taken as validation for an engine that had asserted nothing.

### Describe the behavior you expected

A run that selected no tests would ideally fail, or at least not report success. An opt-out along the lines of `PW_REQUIRE_TESTS` would keep deliberately empty runs possible where they're wanted.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.