apache / apache/datafusion-comet

ci: next set of jobs to move from the PR tier to the merge queue tier

Open
#5,870 1 comment 0 reactions 1 assignee Claimed by @andygrove View on GitHub
area:ci enhancement requires-triage
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

### What is the problem the feature request solves?

#5843 split CI into a PR tier and a merge-queue tier. This issue proposes the next set of tier changes, based on the last 30 days of CI data (every `CI` run on a pull request from Aug 12 to Sep 11, 1,288 runs that actually executed, plus 168 push-to-main runs of the full pipeline used as the flake baseline).

"Unique catches" below means PR runs where the job failed while the Linux build, lint and the Spark 4.1 Linux test profile were all green, i.e. no cheaper job would have flagged the PR.

**Cost and signal of the current PR tier**

| Job | Runner hrs/day | PR fail | Main fail | Unique catches / month | Wall clock |
|---|---|---|---|---|---|
| Spark SQL 4.1, `sql_core-1/2/3` (+ build) | ~96 | 8.5% | 4.5% | 33 | 100 min |
| Spark SQL 4.1, `sql_hive-1/2/3` | ~65 | 1.4% | 1.3% | 7 | 110 min |
| Iceberg 1.11 | 84 | 6.4% | 2.6% | 17 | 100 min |
| Linux tests, Spark 3.4 profile | 46 | 11.9% | 6.0% | 25 | 60 min |
| Linux tests, Spark 3.5 profile | 44 | 11.3% | 6.0% | 20 | 60 min |
| Linux tests, Spark 4.0 profile | 48 | 10.5% | 5.4% | 15 | 60 min |
| Linux tests, Spark 4.2 profile | 45 | 10.0% | 4.8% | 19 | 60 min |
| Linux tests, Spark 4.1 profile | 48 | 9.9% | 4.8% | 58 | 60 min |
| Linux build, lint, rust-test, TPC verify | 58 | 15.3% | 5.8% | 120 | 55 min |
| Delta gate, PyArrow UDF, CodeQL, preflight | ~9 | low | 0% | 33 | under 20 min |

The PR tier costs roughly 540 runner-hours per day. The five Linux test profiles co-fail with each other 70 to 88% of the time. 60% of Iceberg 1.11 failures die at its "Build Comet" step, which the Linux build already catches.

**The queue tier is not healthy enough to take more jobs yet**

On known-good main, the jobs that are now queue-only went red 32% of the time over the month and 40% in September. With `max_entries_to_build: 2` and a 2.5 hour pipeline that eviction rate will stall the queue. Two culprits:

- macOS: red on main 11% of the time (8% in the week of Sep 8). 79 of its 89 unique PR failures are the `[scans]` suite crashing with SIGSEGV in `hdfsThreadDestructor` (#5023, proposed fix #5036, not yet merged). A Sep 9 comment on #5023 reports the same crash on `ubuntu-24.04`.
- Iceberg 1.8/1.9/1.10: red on main 24% of the time in September. About half of that was #5758 (fixed Sep 8); the rest is artifact-store 403/409 errors, Maven connection resets, and a Scala 2.13 Iceberg test flake.

Without macOS and the old Iceberg versions, the queue tier's red rate on main is 5 to 9%.

### Describe the potential solution

In order:

1. **Stabilize the queue tier first.**
- Land #5036 (libhdfs thread destructor use-after-free). Until the crash rate on main is near zero, make macOS `[scans]` label-only rather than queue-tier.
- Track the Iceberg 1.8/1.9/1.10 red rate on main after #5758 and the artifact retry changes (#5842). If it stays above a few percent, move those three to label-only as well.

2. **Move `Spark SQL Tests (Spark 4.1)` `sql_hive-1/2/3` to queue-only.** 65 runner-hours per day for 7 unique catches a month, and it is the 110 minute long pole. Keeps `sql_core-1/2/3` and `catalyst` on PRs, which catch 33 real failures a month across 33 different branches. PR wall clock drops to about 100 minutes. This needs a `hive` on/off input on `spark_sql_test_reusable.yml` and a corresponding POLICY entry in `dev/ci/compute-changes.py`.

3. **Move `Iceberg Spark SQL Tests (Iceberg 1.11)` to queue-only** (opt in on PRs with `run-iceberg-tests`). 84 runner-hours per day for 17 unique catches, which become roughly one queue eviction every two days.

4. **Optionally move the Spark 4.0 and 4.2 Linux test profiles to queue-only.** 93 runner-hours per day for 34 unique catches. Keep 3.4 (Scala 2.12 and the oldest Spark, the most unique catches of the non-4.1 profiles) and 4.1 on PRs. Same mechanism as #5843 for the macOS build: a profile filter input on `pr_build_linux.yml` driven by POLICY.

Steps 2 and 3 save about 150 runner-hours per day (28% of PR spend) for about 24 escapes a month. Adding step 4 brings the saving to about 240 runner-hours per day (45%).

Not recommended: dropping the Spark SQL `sql_core` modules from PRs. They are the best-value heavy job after the Linux build.

### Additional context

- The merge queue ruleset is active but there were zero `merge_group` runs at the time of the analysis (#5843 and #5865 merged just before it took effect), so queue-tier behavior above is inferred from push-to-main runs.
- Unique-catch counts include some flakes; the "Main fail" column is the calibration for that. On main, a flaky failure hits all Linux profiles 80 to 100% of the time, so the per-profile unique catches are mostly real version-specific failures.
- The `.github/workflows/README.md` tier table and diagram need updating with each step.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.