Restore per-version Spark test coverage after the CI trim
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
#19514 leaves one Spark lane on Java 11 and one on Java 17, so the Spark 3.3, 3.4, 4.0 and 4.1 datasource modules are no longer compiled or tested by any job. A change to shared code that updates only the surviving lanes can break them on master with every check green.
#19082 is the intended fix: a curated core test suite on every Spark version, with the full suites reserved for the latest 3.x and 4.x. That is broader version coverage than the pre-trim matrix at a fraction of the cost.
Part of #19524.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading the CI configuration and the linked issues #19514 and #19082 to understand the trimmed matrix and intended curated core suite. Trace the jobs covering the Spark 3.3, 3.4, 4.0 and 4.1 datasource modules, then verify that each version compiles and runs the required tests while full suites remain limited as described.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- ci-cd, data-engineering, testing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 50/100