[CI] Move Spark Scala 2.13 tests from PR CI to nightly
- Dominant language
- Java
- Stars
- 2.1k
- Forks
- 625
- Avg merge
- 3d 14h
- Merged PRs (30d)
- 97
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.
### Description
The Fluss self-hosted CI runners are currently under high load, which causes long queues and increases the time pull requests spend waiting for CI.
The Spark 3 test lanes are among the more expensive jobs. The `spark3` and `spark3-scala213` lanes run the same non-lake Spark test suite; the main difference is that the latter switches the Scala version from 2.12 to 2.13. Scala-version-specific regressions are relatively uncommon, so running both mirrored lanes on every pull request has a high runner-cost-to-signal ratio.
A recent PR run shows the current cost:
- [`spark3`](https://github.com/apache/fluss/actions/runs/31360307617/job/93367748148?pr=3831): 28 minutes
- [`spark3-lake`](https://github.com/apache/fluss/actions/runs/31360307617/job/93367748124?pr=3831): 36 minutes
- [`spark3-scala213`](https://github.com/apache/fluss/actions/runs/31360307617/job/93367748127?pr=3831): 28 minutes
Related work:
- #3273 proposes representative-version coverage to reduce duplicated Flink multi-version CI tests.
- #3717 split the Spark lake tests from the two Scala lanes, but intentionally kept `spark3-scala213` as a non-lake compatibility lane on every PR. It has been merged, and recent runs still show the Scala 2.13 lane taking about 26–28 minutes.
We should move the `spark3-scala213` compatibility lane from the default Java 11 pull-request CI to the scheduled Nightly workflow. The Nightly workflow already calls the shared CI template, so the workflow configuration should be parameterized (or otherwise separated) to ensure that:
- pull-request CI keeps `spark3` and `spark3-lake`, but does not schedule `spark3-scala213` by default;
- Nightly continues to run the full `spark3-scala213` test lane;
- the same Scala 2.13 test selection currently used by PR CI is preserved;
- Scala 2.13 coverage can still be run manually when needed.
This removes one roughly 28–30 minute self-hosted runner job from every pull request. Although matrix jobs can run in parallel when capacity is available, reducing this duplicated runner load should shorten queueing and end-to-end CI waiting time under the current constrained capacity.
The trade-off is that a Scala 2.13-specific regression may be detected by Nightly rather than before merge. Given that the test suite mirrors the primary Scala 2.12 lane and Scala-version-specific failures are uncommon, this seems like a reasonable trade-off. As an optional safeguard, we can still trigger the lane for pull requests that modify Scala-version profiles or related Spark build configuration.
### Willingness to contribute
- [ ] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by locating the Nightly workflow, the shared CI template, and the configuration for the spark3, spark3-lake, and spark3-scala213 lanes. Compare how PR and Nightly workflows select the Scala 2.13 tests, then verify that PR CI omits that lane while Nightly and manual runs retain it with the existing test selection.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github-actions, scala, spark
- Domain
- build-system, ci-cd
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100