apache / apache/hudi

Cut CI wall clock: rebalance the Spark test jobs and surface the slow tail

Open
#19,528 0 comments 0 reactions 1 assignee Claimed by @voonhous View on GitHub
area:ci priority:high
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

The critical path for a PR is `test-spark-java17-java-tests-part1` at roughly 70 minutes, and the part1/part2/part3 splits follow module boundaries rather than measured duration.

- Rebalance the splits by measured job duration, and re-measure whenever what those jobs run changes.
- Report the slowest tests per job so the long tail is visible and can be fixed or moved off the PR path.
- Give contributors early signal: run the cheap validations and a fast compile before the long test jobs, so an obviously broken PR fails in minutes rather than after an hour.

Part of #19524.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.