[CI] Java CI runner-minutes and critical runtime: PR tracker

Part of #19524. Baseline is master run [34314808833](https://github.com/apache/hudi/actions/runs/34314808833): 615 runner-minutes over 19 test jobs, longest job 77 min, two trips through the runner queue per run.

**Scope.** The two metrics tracked here are execution time only: total runner-minutes of the 19 test jobs, and the longest job's execution time. Queue time is not the problem being solved now, and workflow changes are kept minimal. #19890 (flattening the job graph) removes a second trip through the runner queue, which is outside these metrics, so it is not justified by this scorecard; it stays listed because #19892 and #19909 were built on top of it.

**Reading the numbers.** Each job runs on its own 4-vCPU runner whose speed varies by draw: on a slow machine every test class and the build step stretch by the same factor, 10 to 30%, and on a fast one they shrink. Untouched jobs therefore move by up to 10 minutes between attempts of the same commit, and the run total by up to 35 (#19919: 539, 531, 551, 516 over four attempts, no code change). So a single run total can only judge changes of #19892's size, around 40 runner-minutes or more. Anything smaller is judged on the steps of the job it touches, across at least three attempts, and counts only if the direction is the same every time. The critical-runtime column has the same floor: the longest job moves 39 to 44 between attempts on its own.

**Scorecard**, measured runs, cumulative down the stack (each column includes everything to its left). Runner-minutes are the sum of the 19 test jobs' execution times; critical runtime is the longest job; both exclude queue time.

| metric | master baseline | + #19890 | + #19892 | + #19909 (three samples: its two attempts and #19918's run, which adds no work) | + #19919 (four samples: its four attempts) | + #19926, #19927, #19928 (two samples so far: #19928's two runs) | + #19932 (three samples) | + #19931 and #19933 = end state, measured on #19937 (three samples) |
|---|---|---|---|---|---|---|---|---|
| total runner-minutes per run | 615 | 617 | 577 | 544, 558, 541; mean 548 (-11% vs baseline, -5% vs #19892) | 539, 531, 551, 516; mean 534 (-1% vs #19918's 541, inside noise: the attempts differ by 35 among themselves; integration-tests 37 -> 28, 25, 27, 28 min) | [534](https://github.com/apache/hudi/actions/runs/34708740840), [509](https://github.com/apache/hudi/actions/runs/34711686507), [489](https://github.com/apache/hudi/actions/runs/34711686507/attempts/3), [521](https://github.com/apache/hudi/actions/runs/34711686507/attempts/5), [507](https://github.com/apache/hudi/actions/runs/34742475247): mean 512 (-4% vs #19919's 534); the three parallel shards cost 82, 78, 72, 80 and 76 runner-minutes against 109 serial, about -30 per run | [514](https://github.com/apache/hudi/actions/runs/34748622752), [491](https://github.com/apache/hudi/actions/runs/34748622752/attempts/2), [512](https://github.com/apache/hudi/actions/runs/34748622752/attempts/3): mean 506; other-2's 33 shared-session suites 30 to 31 min as a job against 35 to 37 | [511](https://github.com/apache/hudi/actions/runs/34753265152), [506](https://github.com/apache/hudi/actions/runs/34753265152/attempts/2), [491](https://github.com/apache/hudi/actions/runs/34753265152/attempts/3): mean 503 (-18% vs baseline); 18 test jobs |
| critical runtime (longest job) | 77 min | 74 min | 41 min | 41, 44, 40: unchanged. flink-2 dropped from 39 to 23, but the Scala shards (scala-other-1, scala-dml-1) were already at about 40 and now set the wall clock | 40, 41, 40, 39 min: unchanged (also [41](https://github.com/apache/hudi/actions/runs/34720098561) on #19931, the same layout plus job timeouts) | [37](https://github.com/apache/hudi/actions/runs/34708740840), [36](https://github.com/apache/hudi/actions/runs/34711686507), [35](https://github.com/apache/hudi/actions/runs/34711686507/attempts/3), [37](https://github.com/apache/hudi/actions/runs/34711686507/attempts/5), [37](https://github.com/apache/hudi/actions/runs/34742475247) min: the Scala shards leave the critical path (dml-1 28, dml-2 20 to 24, other-1 30 as jobs); scala-other-2, fta and ut-other now set it at 32 to 37 | [35](https://github.com/apache/hudi/actions/runs/34748622752), [33](https://github.com/apache/hudi/actions/runs/34748622752/attempts/2), [39](https://github.com/apache/hudi/actions/runs/34748622752/attempts/3) (fta, fta, ut-fat-and-ftc): other-2 leaves the critical path too; the Java pack (fta, spark-client, ut-other, common-and-other, ut-fat-and-ftc) at 32 to 35 is the floor for the Scala work | [37](https://github.com/apache/hudi/actions/runs/34753265152), [37](https://github.com/apache/hudi/actions/runs/34753265152/attempts/2), [35](https://github.com/apache/hudi/actions/runs/34753265152/attempts/3) min (-55% vs 77): ut-fat-and-ftc, ut-fat-and-ftc, fta |
| queue trips per run | 2 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |

Runs: #19919 [34670498928](https://github.com/apache/hudi/actions/runs/34670498928) and [34670498897](https://github.com/apache/hudi/actions/runs/34670498897) (attempt 1: 539 min, longest 40; [attempt 2](https://github.com/apache/hudi/actions/runs/34670498928/attempts/2): 531, longest 41; [attempt 3](https://github.com/apache/hudi/actions/runs/34670498928/attempts/3): 551, longest 40; [attempt 4](https://github.com/apache/hudi/actions/runs/34670498928/attempts/4): 516, longest 39; integration-tests 37 -> 28, 25, 27, 28); #19918 [34668461677](https://github.com/apache/hudi/actions/runs/34668461677) and [34668461644](https://github.com/apache/hudi/actions/runs/34668461644) (third sample of the #19909 stack: 541 min, longest 40); baseline [34314808833](https://github.com/apache/hudi/actions/runs/34314808833); #19890 [34469837248](https://github.com/apache/hudi/actions/runs/34469837248) and [34469836734](https://github.com/apache/hudi/actions/runs/34469836734); #19892 [34499656524](https://github.com/apache/hudi/actions/runs/34499656524) and [34499656559](https://github.com/apache/hudi/actions/runs/34499656559); #19909 [34637051799](https://github.com/apache/hudi/actions/runs/34637051799) and [34637051750](https://github.com/apache/hudi/actions/runs/34637051750).

| PR | change | runner-min per run | critical runtime | status |
|---|---|---|---|---|
| #19890 | flatten the job graph, path filter at the trigger | 615 -> 617 (unchanged) | execution unchanged; removes the second queue trip (median 1 min, p90 82 min), which is outside the tracked metrics | measured; not justified by the scorecard, kept only as the base of #19892 and #19909 |
| #19892 | two surefire forks, DeltaStreamer split, jobs resharded by measured time | 615 -> 577 | longest job 77 -> 41 min | measured, ready to merge after #19890 |
| #19906 | test Spark sessions at 2 shuffle partitions, AQE off | 447 -> 426 on the Spark lanes, mean of three runs (-5%), while the untouched engine jobs drifted +19% | none measurable (same job varies 26 to 42 min between runs) | closed, not merged: gain inside runner noise, fta consistently +6 min; the local A/B that motivated it did not reproduce. Findings: https://github.com/apache/hudi/pull/19906#issuecomment-5637694386 and https://github.com/apache/hudi/pull/19906#issuecomment-5638151239. It also exposed #19911, fixed separately |
| #19909 | one full build per module and lane: flink-2 filter, docker build trim, examples-spark trim | 577 -> 544 and 558 (mean -26, -4.5%); build steps 5829 -> ~5200 s | flink-2 39 -> 23, docker job 37 -> 28; longest job unchanged at 41 to 44, now the Scala shards | measured twice on the same commit; ready to merge after #19892 |
| #19918 | slowest tests per job on the run page (closes #19528) | none (a reporting step, under a second); its run measured 541 for the stack | none; longest job 40 (scala-dml-1); makes the long tail visible for the next rebalance | green on [34668461677](https://github.com/apache/hudi/actions/runs/34668461677) and [34668461644](https://github.com/apache/hudi/actions/runs/34668461644), tables on every test job's summary; ready to merge after #19909 |
| #19919 | integration-tests builds only the 30-module closure of its four IT modules and verifies just those four (was 66 built, 64 verified) | 541 -> 539, 531, 551, 516 over four attempts (mean 534, -1%; the attempts differ by 35 among themselves, so the total cannot resolve it); the job itself 37 -> 28, 25, 27, 28 min (build step 646 -> 479, 456, 515, 495 s; verify step 1390 -> 1028, 971, 985, 1006 s), all four below every one of about 90 earlier samples of that job, whose verify step never ran under 1258 s | none; longest job 40, 41, 40, 39 min, the Scala shards still set the wall clock | measured four times ([attempt 1](https://github.com/apache/hudi/actions/runs/34670498928), [attempt 2](https://github.com/apache/hudi/actions/runs/34670498928/attempts/2), [attempt 3](https://github.com/apache/hudi/actions/runs/34670498928/attempts/3), [attempt 4](https://github.com/apache/hudi/actions/runs/34670498928/attempts/4)): same 25 test classes (14 IT) and the same per-module totals in the IT step's log; ready to merge after #19918 |
| #19926, #19927, #19928 (successors of #19921, #19923, #19925, stacked on #19919) | Scala SQL suites share one SparkContext per JVM (opt-in), cross-suite state removed, then dml-1, dml-2 and other-1 run suites two at a time (scalatest `-P2`) | the three shards 109 -> 82 and 78 runner-minutes per run (-27); run totals [534](https://github.com/apache/hudi/actions/runs/34708740840), [509](https://github.com/apache/hudi/actions/runs/34711686507), [489](https://github.com/apache/hudi/actions/runs/34711686507/attempts/3), [521](https://github.com/apache/hudi/actions/runs/34711686507/attempts/5) and [507](https://github.com/apache/hudi/actions/runs/34742475247) (mean 512) against #19919's 516 to 551 (mean 534) | dml-1 30:28 to 30:52 -> 15:25 to 20:19 over six samples (-34% to -50%); dml-2 18:46 to 23:01 -> 13:27 to 16:56 over five (-23%); other-1 31:52 to 33:06 -> 17:14 to 22:34 over four (-36%); scalatest wall clocks, same test names as the serial runs; longest job 41 -> [37](https://github.com/apache/hudi/actions/runs/34708740840), [36](https://github.com/apache/hudi/actions/runs/34711686507), [35](https://github.com/apache/hudi/actions/runs/34711686507/attempts/3), [37](https://github.com/apache/hudi/actions/runs/34711686507/attempts/5), [37](https://github.com/apache/hudi/actions/runs/34742475247), now the serial scala-other-2 (until #19932) | #19926 and #19927 green; #19928 measured four times at two threads (verdict on its description) and twice at four threads ([run 34737535334](https://github.com/apache/hudi/actions/runs/34737535334), [attempt 2](https://github.com/apache/hudi/actions/runs/34737535334/attempts/2): dml-1 17:20, 13:35; dml-2 13:17, 14:31; other-1 20:04, 18:43; longest job 36, 36; totals 499, 520); other-1's first run exposed a shared-context leak in ParquetUtils.readAvroRecords (fixed in #19926) and a catalog-wide show tables assertion (fixed in #19927); next: four threads, then a reshard that moves the SQL-base suites out of scala-other-2 |
| #19932 (on #19928) | scala-other-2 runs its 33 HoodieSparkSqlTestBase suites shared and two at a time through an inherited ScalaTest tag; its 10 own-context suites (2 min) run as a step in ftb | run totals [514](https://github.com/apache/hudi/actions/runs/34748622752), [491](https://github.com/apache/hudi/actions/runs/34748622752/attempts/2) and [512](https://github.com/apache/hudi/actions/runs/34748622752/attempts/3) (mean 506) against 489 to 534 | scala-other-2 tests 28:24 -> [23:34](https://github.com/apache/hudi/actions/runs/34748622752/job/103700994798), [23:44](https://github.com/apache/hudi/actions/runs/34748622752/attempts/2/job/103703547405), [22:56](https://github.com/apache/hudi/actions/runs/34748622752/attempts/3/job/103707769911) (job 35 to 37 -> 31, 31, 30); ftb 21 to 28 -> [30](https://github.com/apache/hudi/actions/runs/34748622752/job/103700994761), [30](https://github.com/apache/hudi/actions/runs/34748622752/attempts/2/job/103703547479), [35](https://github.com/apache/hudi/actions/runs/34748622752/attempts/3/job/103707769878) (its new step 2:26 to 2:49 plus a minute of Maven); longest job [35](https://github.com/apache/hudi/actions/runs/34748622752), [33](https://github.com/apache/hudi/actions/runs/34748622752/attempts/2), [39](https://github.com/apache/hudi/actions/runs/34748622752/attempts/3): fta, fta, ut-fat-and-ftc | measured three times, verdict on its description: the Scala shards no longer set the wall clock; its first runs exposed two hudi-hadoop-common stream-close gaps, split out as #19935 and #19936; other-2's floor is its exclusive suites (1.2 threads busy) |
| #19931 (on #19919) | `timeout-minutes: 75` on the nine Java 17 Spark jobs and the five engine jobs, which had none | none ([555](https://github.com/apache/hudi/actions/runs/34735853720) on its run, serial Scala shards) | none; a hung job now fails at 75 min instead of GitHub's 6 h | green; motivated by a surefire fork hang that idled 76 min on #19928 and a flink-1 hang that ran the full 6 h on this PR's own first run |
| #19933 (on #19931) | hadoop-mr and java-client tests folded into utilities-rest: one JDK 11 job fewer, one build fewer | [542](https://github.com/apache/hudi/actions/runs/34743931958) against 555 on #19931 (one sample); the merged job [30 min](https://github.com/apache/hudi/actions/runs/34743931958/job/103688224470) against 22 plus 10 as two jobs | none; longest job 40 on both runs (serial Scala shards, this branch has no Scala change) | green; Java CI 14 -> 13 jobs; the only fold that stays under the longest jobs, every other pair lands on or above them |
| #19935, #19936 (off master) | two hudi-hadoop-common stream-close fixes that #19932's first runs exposed through Spark's DebugFilesystem: the rebuffered stream never closed the filesystem's own stream object (accounting), and the inline filesystem never closed the outer file of an embedded HFile read (a real handle leak on every metadata table log block read) | none | none | open, each with a unit test; #19932 carries copies until they merge |
| #19937 (everything above, DNM) | measurement only: #19932 plus #19931 plus #19933 on one branch, never merged | [511](https://github.com/apache/hudi/actions/runs/34753265152), [506](https://github.com/apache/hudi/actions/runs/34753265152/attempts/2), [491](https://github.com/apache/hudi/actions/runs/34753265152/attempts/3): mean 503, -18% vs the 615 baseline | [37](https://github.com/apache/hudi/actions/runs/34753265152), [37](https://github.com/apache/hudi/actions/runs/34753265152/attempts/2), [35](https://github.com/apache/hudi/actions/runs/34753265152/attempts/3): ut-fat-and-ftc twice, then fta and spark-client tied; -55% vs 77 | its table holds the campaign's end state; closed after the samples |
| #19905 | Maven build cache | - | - | closed: the repository's Actions cache store keeps nothing |

Measured once #19890, #19892, #19909 and #19919 merge: about 615 -> 540 runner-minutes (-12%), longest job 77 -> about 41 min (-47%), one queue hop. Getting the longest job below 40 means the Scala shards: scalatest has no fork count, so that is a re-split of the suites or faster tests, not a build trim.

Merge order: the checklist in [this comment](https://github.com/apache/hudi/issues/19910#issuecomment-5658468456).

Findings on record:
- test-count parity, master against the end state, from every job log of [master run 34674848405](https://github.com/apache/hudi/actions/runs/34674848405) and [#19937's attempt 3](https://github.com/apache/hudi/actions/runs/34753265152/attempts/3) plus its engine run: ScalaTest identical, 963 passed, 5 canceled, 3 ignored on both sides and the same 986 test names. Surefire 19267 tests on master against 17470 on the end state (Trino plugin excluded, its run is not part of master's Java CI), 0 failures on both, and every difference has a cause: master ran the 113 hudi-flink test classes twice, in flink-2 and again in common-and-other (1808 duplicate runs, removed by #19909); the utilities tests moved from common-and-other into the two utilities jobs (1718 against 1721 tests, same classes); TestHoodieDeltaStreamer is four classes now (197 against 199); the bloom filter benchmark is out of unit runs (1 test, by design in #19892); two new unit tests from #19935 and #19936; and a few classes whose methods surefire reports across its two forks as two partial summaries. No class runs fewer tests on the end state than on master.
- scalatest threads, two against four on the three parallel SQL shards of #19928 ([run 34737535334](https://github.com/apache/hudi/actions/runs/34737535334), [attempt 2](https://github.com/apache/hudi/actions/runs/34737535334/attempts/2)): four threads keep 3.6 of them busy on dml-1 and dml-2 and 3.0 on other-1, but every suite runs about 60% longer, since four Spark sessions share four vCPUs. The shard wall clock gains only on dml-1 (17:20, 13:35 against 19:36 to 20:19 at two threads) and is flat on dml-2 (13:17, 14:31 against 13:27 to 16:56) and other-1 (20:04, 18:43 against 17:14 to 22:34). Run totals 499 and 520, longest job 36 both times, all six shard runs green with the same test names. Kept at two: the gain sits on a shard that does not set the wall clock, and four narrows every suite's timing margin. Four is a measured, safe setting for a shard that becomes the critical path.

Deferred, with reasons on record:
- method-level test concurrency: https://github.com/apache/hudi/pull/18201#issuecomment-5633539153
- path scoping between the Spark and engine workflows (~11% of runner-minutes, needs a path-to-lane map)
- pinning six transitive version ranges (junit, junit-dep, json-smart, grpc x3) to remove metadata checks against jitpack.io and cloudera; one 16-minute stall observed
- running test-common-and-other-modules outside its 2024 container (maintenance, not time)