rstudio / rstudio/rstudio

e2e: scheduled rotation tests a stale daily because /rstudio/latest/index.json lags the branch rollover

Open
#18,498 0 comments 0 reactions 0 assignees View on GitHub
bug builds
Dominant language
Java
Stars
5.1k
Forks
1.2k
PR merge metrics
PR metrics pending

Description

## Summary

The scheduled e2e rotation runs the tests from `main` against a daily build from the
`rel-yellow-yarrow` (2026.08) branch. Any test that asserts behavior added to `main`
after the 2026.08 branch cut fails on every platform, because the binary under test
does not contain the code.

This is not a test defect and not flake. The tests are correct and the product is
correct. Only the pairing is wrong.

## Symptom

Scheduled runs [31303035336](https://github.com/rstudio/rstudio/actions/runs/31303035336)
(2026-08-09) and [31371013310](https://github.com/rstudio/rstudio/actions/runs/31371013310)
(2026-08-10) failed the same tests on Ubuntu 24, Rocky 9, Rocky 10, Ubuntu 26,
macOS 14, macOS 15, macOS 26, and Windows Server 2025:

- `tests/panes/layout/panes.test.ts` -- 8 of the 9 tests added by #18454
- `tests/panes/editor/cpp_chunk_indent.test.ts` -- the test added by #18470

A representative failure:

```
panes/layout/panes.test.ts:760 › Restoring a zoomed pane from its header button ends the zoom (#18444)

Error: zoom bookkeeping should be cleared after escaping the zoom
Expected: false
Received: true
```

That is exactly the pre-#18454 behavior. The pane header button restored the vertical
split through `DualWindowLayoutPanel` but left `PaneManager.maximizedTab_` set.
`hookPaneMaximize` (`src/gwt/src/org/rstudio/studio/client/workbench/ui/PaneManager.java:1344`)
is the code that routes the gesture through `restoreLayout()`, and it is absent from
the build under test.

The failure set confirms the diagnosis. #18454 added 9 tests, 8 fail, and the only
one that passes -- `'The pane header button still maximizes when nothing is zoomed'`
-- is the one that asserts behavior which predates the fix.

## Root cause

The rotation installs the build named by `https://dailies.rstudio.com/rstudio/latest/index.json`.
That JSON object is stale. The HTML page at the same path is not:

| URL | `last-modified` | Serves |
| --- | --- | --- |
| `/rstudio/latest/` (page) | Mon, 10 Aug 2026 19:36 GMT | Autumn Hawkbit, `2026.09.0-daily+11` |
| `/rstudio/latest/index.json` | **Sat, 08 Aug 2026 05:33 GMT** | Yellow Yarrow, `2026.08.0-daily+186` |
| `/rstudio/autumn-hawkbit/index.json` | Mon, 10 Aug 2026 18:32 GMT | Autumn Hawkbit, `2026.09.0-daily+11` |

So the `latest` alias rolled over to `main` for browsers but not for the JSON manifest
that CI consumes. The per-branch manifest is current and correct.

Commit check:

```
build installed by the rotation: 2026.08.0-daily+186 -> 0731c0d19d / a249c8fa0c (rel-yellow-yarrow)
build the page advertises: 2026.09.0-daily+11 -> 2f648f3a0d (main)

8b3b385e55 (#18454, pane zoom) NOT in a249c8fa0c IS in 2f648f3a0d
7ed4e560a0 (#18470, cpp chunk indent) NOT in a249c8fa0c IS in 2f648f3a0d
```

Reproduce:

```bash
curl -sSI https://dailies.rstudio.com/rstudio/latest/index.json | grep -i last-modified
curl -fsSL https://dailies.rstudio.com/rstudio/latest/index.json | jq -r '.branch, .version'
curl -fsSL https://dailies.rstudio.com/rstudio/autumn-hawkbit/index.json | jq -r '.branch, .version'
```

## Why PR runs are unaffected

`os-test-e2e-rstudio-pr.yml` already guards against this. Its `decide()` function
requires the daily's commit to contain the PR's merge base, and falls back to a source
build when it does not. #18454 and #18470 both touched `src/`, so both built from
source and went green. The guard fails closed, which is why the problem only shows up
in the rotation.

`os-test-e2e-rstudio-scheduled.yml` has no equivalent check. It installs whatever
`latest/index.json` names and runs `main`'s tests against it.

## Blast radius

Nine workflows read the stale manifest:

```
.github/workflows/os-test-e2e-rstudio-desktop-linux.yml
.github/workflows/os-test-e2e-rstudio-desktop-os-autocomplete-scheduled-windows-server-2025.yml
.github/workflows/os-test-e2e-rstudio-desktop-os-macos-14-arm64.yml
.github/workflows/os-test-e2e-rstudio-desktop-os-macos-15-x86_64.yml
.github/workflows/os-test-e2e-rstudio-desktop-os-macos-26-arm64.yml
.github/workflows/os-test-e2e-rstudio-desktop-os-windows-server-2025.yml
.github/workflows/os-test-e2e-rstudio-pr.yml
.github/workflows/os-test-e2e-rstudio-scheduled.yml
.github/workflows/os-test-e2e-rstudio-server-os-ubuntu-24.yml
```

The Windows one uses PowerShell (`Invoke-RestMethod`); the rest use `curl` + `jq`.

## Potential fixes

### 1. Resolve the manifest per branch (recommended)

Derive the manifest path from `version/RELEASE`, which is already in the workspace by
the time the installer is resolved. Lowercase it and replace the space with a hyphen:
`Autumn Hawkbit` -> `autumn-hawkbit`, `Yellow Yarrow` -> `yellow-yarrow`. Both paths
exist and both are current.

```bash
BRANCH_SLUG=$(tr 'A-Z ' 'a-z-' < version/RELEASE)
MANIFEST_URL="https://dailies.rstudio.com/rstudio/${BRANCH_SLUG}/index.json"
if ! MANIFEST=$(curl -fsSL --retry 3 --retry-delay 5 "$MANIFEST_URL"); then
echo "::error::No daily manifest for branch '$BRANCH_SLUG' at $MANIFEST_URL."
exit 1
fi
```

This stops depending on the `latest` alias, so a stale alias cannot mis-pair the build
again. It also pairs a release-branch PR with that branch's daily, and it removes the
needless source builds the PR workflow does today whenever `latest` lags.

Cost: nine files, one of them PowerShell. Worth factoring the resolution into a single
composite action in the same change.

### 2. Add the merge-base staleness guard to the rotation

Port `decide()` from `os-test-e2e-rstudio-pr.yml` into
`os-test-e2e-rstudio-scheduled.yml`, so a daily that does not contain `main`'s HEAD
triggers a source build.

Correct, and it makes the rotation self-defending against any future skew. But while
the manifest is stale it would build every engine from source -- more than ten of
them -- which is a large cost for a rotation whose stated purpose is to test the
shipped daily.

Best used together with option 1, as a backstop rather than the primary fix.

### 3. Fix the publisher

`/rstudio/latest/index.json` should be regenerated whenever `/rstudio/latest/` is.
This needs whoever owns the dailies site, so it is not fixable in this repo. Even
after it is fixed, option 1 is still worth doing: a consumer that names its own branch
cannot be mis-pointed by an alias.

### Not recommended: skip or version-gate the tests

Adding `test.skip` or a version guard to the affected specs would turn a build-pairing
problem into permanently disabled coverage, and would hide the next occurrence.

## Suggested order

1. Report the stale `index.json` to the dailies owners (option 3).
2. Land option 1 so CI stops depending on `latest`.
3. Land option 2 as a backstop.

Contributor guide

Open the contributing guide

Research direction

Start with .github/workflows/os-test-e2e-rstudio-scheduled.yml and compare its daily resolution with decide() in os-test-e2e-rstudio-pr.yml. Review the other listed workflow files, including the PowerShell Windows workflow, and determine how the branch-specific manifest should be resolved or factored into a composite action. Done means scheduled workflows no longer depend on the stale latest manifest and continue to install the matching daily.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, powershell, shell
Domain
ci-cd, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.