apache / apache/hudi

Restore per-version Spark test coverage after the CI trim

Open
#19,531 0 comments 0 reactions 0 assignees View on GitHub
area:ci area:tests priority:high
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

#19514 leaves one Spark lane on Java 11 and one on Java 17, so the Spark 3.3, 3.4, 4.0 and 4.1 datasource modules are no longer compiled or tested by any job. A change to shared code that updates only the surviving lanes can break them on master with every check green.

#19082 is the intended fix: a curated core test suite on every Spark version, with the full suites reserved for the latest 3.x and 4.x. That is broader version coverage than the pre-trim matrix at a fraction of the cost.

Part of #19524.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading the CI configuration and the linked issues #19514 and #19082 to understand the trimmed matrix and intended curated core suite. Trace the jobs covering the Spark 3.3, 3.4, 4.0 and 4.1 datasource modules, then verify that each version compiles and runs the required tests while full suites remain limited as described.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
ci-cd, data-engineering, testing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
50/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.