SeleniumHQ / SeleniumHQ/selenium

[🚀 Feature]: Selenium GHA Testing Plan

Open
#17,539 1 comment 0 reactions 0 assignees View on GitHub
A-needs decision I-enhancement
Dominant language
Java
Stars
34.5k
Forks
8.7k
Avg merge
2d 1h
Merged PRs (30d)
92

Description

### Description

This supersedes #16809 - a lot of underlying work has been done on build process across the bindings since that issue was created to support a slightly better vision.

### Problem

* Github Actions runs are taking too long, especially when multiple small PRs are being made.
* ci.yml workflow has significant overlap with ci-rbe.yml workflow
* There are things that can't be tested in RBE that aren't being tested on a GHA Runner
* Tests are inconsistent across bindings

### Proposal

RBE remains the primary per-PR test gate. GHA Runners should only run tests that provide sufficient additional signal.

GHA runners should be scoped by affected Bazel targets (this is wired up but not used yet).

#### GHA Tests

**Update**: I think we move all browser tests that aren't se-manager or os-sensitive to manual/scheduled; Same with unit test matrix; we should trust that RBE will find enough of the unit test issues that it isn't worth rerunning every combination on every PR.

**Run every time there are affected targets:**
1. Windows: Affected targets list filtered for ~Edge~ + se_manager + os_sensitive ~-safari -ie~ (large & small tests)
2. Mac: Affected targets list filtered for ~Safari~ + se_manager + os_sensitive ~-ie~ (large & small tests)
3. Linux: Affected targets list filtered for se_manager + rbe_skipped ~-safari -ie~
~Run when tagged with [language]~
~4. Unit Test Matrix: first/last version and alternate interpreters as applicable~

**Run manually and/or scheduled:**
1. Additional unit tests for alternate interpreters or other versions
2. **Updated:** Windows: chrome-beta, firefox-beta, edge (local)
3. **Updated:** Mac: safari (local)

### Additional/Future Considerations
1. consider running periodically all targets on windows not filtering to just edge as a stopgap
2. ~Ideally we can replace .skipped-tests with bazel tags or test runner guards/annotations~ **Update**: Done
3. ~fix Java to use target_compatible_with instead of generating targets for everything and filtering~ **Update**: Done
4. consider splitting skip-rbe tag into a reason for skipping
5. consider running scheduled ci.yml workflow 1x a day instead of 2x a day. We've averaged 4 commits a day on trunk for the past year; 90% of days are < 9 commits. We run this much more frequently than we evaluate it.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.