SeleniumHQ / SeleniumHQ/selenium
[🚀 Feature]: Selenium GHA Testing Plan
- Dominant language
- Java
- Stars
- 34.5k
- Forks
- 8.7k
- Avg merge
- 2d 1h
- Merged PRs (30d)
- 92
Description
### Description
This supersedes #16809 - a lot of underlying work has been done on build process across the bindings since that issue was created to support a slightly better vision.
### Problem
* Github Actions runs are taking too long, especially when multiple small PRs are being made.
* ci.yml workflow has significant overlap with ci-rbe.yml workflow
* There are things that can't be tested in RBE that aren't being tested on a GHA Runner
* Tests are inconsistent across bindings
### Proposal
RBE remains the primary per-PR test gate. GHA Runners should only run tests that provide sufficient additional signal.
GHA runners should be scoped by affected Bazel targets (this is wired up but not used yet).
#### GHA Tests
**Update**: I think we move all browser tests that aren't se-manager or os-sensitive to manual/scheduled; Same with unit test matrix; we should trust that RBE will find enough of the unit test issues that it isn't worth rerunning every combination on every PR.
**Run every time there are affected targets:**
1. Windows: Affected targets list filtered for ~Edge~ + se_manager + os_sensitive ~-safari -ie~ (large & small tests)
2. Mac: Affected targets list filtered for ~Safari~ + se_manager + os_sensitive ~-ie~ (large & small tests)
3. Linux: Affected targets list filtered for se_manager + rbe_skipped ~-safari -ie~
~Run when tagged with [language]~
~4. Unit Test Matrix: first/last version and alternate interpreters as applicable~
**Run manually and/or scheduled:**
1. Additional unit tests for alternate interpreters or other versions
2. **Updated:** Windows: chrome-beta, firefox-beta, edge (local)
3. **Updated:** Mac: safari (local)
### Additional/Future Considerations
1. consider running periodically all targets on windows not filtering to just edge as a stopgap
2. ~Ideally we can replace .skipped-tests with bazel tags or test runner guards/annotations~ **Update**: Done
3. ~fix Java to use target_compatible_with instead of generating targets for everything and filtering~ **Update**: Done
4. consider splitting skip-rbe tag into a reason for skipping
5. consider running scheduled ci.yml workflow 1x a day instead of 2x a day. We've averaged 4 commits a day on trunk for the past year; 90% of days are < 9 commits. We run this much more frequently than we evaluate it.
Contributor guide
Assessment
This issue has not been assessed yet.