[Umbrella][CI] Redesign PR validation for fast, trusted, and scalable feedback

## Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct).

## Search before asking

- [x] I searched the existing issues and pull requests and found no umbrella issue covering the complete PR CI architecture described below.

Related work exists, but covers narrower parts of the problem:

- #9976 extracts long-running modules and improves scheduling/free-disk handling.
- #11074 tracks one source of unrelated modules entering incremental IT builds.
- #9900 previously improved job ordering, timeout selection, and CI duration analysis.

This proposal is intended to coordinate and build on that work, not replace it.

## Summary

SeaTunnel PR validation currently combines broad static fan-out, limited fork-level runner concurrency, repeated builds, weak cross-job failure propagation, and an eventually-consistent status mirror. The result is slow and sometimes misleading feedback for contributors, especially for changes under `seatunnel-engine/**`.

The proposed target is a trusted, risk-tiered CI pipeline:

1. fast required checks on every PR;
2. changed-module and impact-based unit/integration tests;
3. a small engine/connector canary suite for medium-risk changes;
4. exhaustive OS/JDK/connector coverage in a merge queue, explicit `full-ci` run, and nightly validation;
5. build artifacts once per SHA/JDK and reuse them across E2E jobs;
6. direct and accurate check reporting from trusted workflow definitions.

The goal is to reduce normal PR feedback time and runner consumption without removing exhaustive validation from high-risk changes and merge candidates.

## Evidence

### Method and limitations

The following snapshot was collected on 2026-07-21. The sampling frame contained 530 PRs created between 2026-05-01 and 2026-07-21, within the normal Actions retention window. Two hundred PRs were selected at approximately uniform intervals over creation time rather than taking only the newest or currently active PRs. The upstream `Build` check was then followed to the actual workflow run and expanded job list in the contributor fork.

- 185 of 200 sampled PRs resolved to actual runs, covering 35 authors and 37 distinct contributor forks. Fifteen had no usable Build pointer or an unavailable fork run and remain visible as missing data rather than being counted as zero-duration builds.
- Scope classification uses changed file paths. The API returned the first 100 paths; only two sampled PRs exceeded 100 changed files, so their classification is potentially incomplete.
- `Expanded jobs` includes matrix entries that GitHub later marks skipped. `Runnable jobs` excludes skipped entries; the two numbers must not be conflated.
- Attempt wall-clock uses `run_started_at` to `updated_at`. Fifty-two of 183 runs with attempt metadata had `run_attempt > 1`; using the original run `created_at` would incorrectly include hours or days between manual reruns. Performance baselines below therefore use `run_attempt == 1`. Reruns are analyzed separately because GitHub may rerun only failed jobs, making a later attempt faster than the original full validation.
- An earlier 30-PR draft used `created_at` to `updated_at` and therefore overstated three rerun examples as 28-30 hour builds. Those values are removed in this revision. Their latest attempts actually took approximately 108, 203, and 177 minutes.
- Job-level runner-hours are intentionally omitted. Retries, skipped jobs, and invalid job intervals make them unreliable without attempt-aware telemetry. Job counts, attempt wall-clock, workflow structure, and concrete logs are the stronger evidence here.
- This remains an observational sample rather than a controlled benchmark. It exposes architectural failure modes, but long-term SLOs should be based on continuously collected, attempt-aware CI telemetry.

### Cross-contributor results

| Metric | Result |
| --- | ---: |
| Sampling frame / selected / valid runs | 530 / 200 / 185 |
| Authors / distinct forks | 35 / 37 |
| Expanded jobs | min 47, median 61, p90 83, max 84 |
| Runnable jobs | min 3, median 28, p90 75, max 76 |
| Run conclusions at collection time | 132 success, 31 failure, 16 cancelled, 6 active |
| First attempts / timed completed first attempts | 131 / 113 |
| First-attempt conclusions | 92 success, 21 failure, 13 cancelled, 5 active |
| First-attempt wall-clock | median 82 min, p75 220 min, p90 454 min, p95 620 min, max 985 min |

The original four-run snapshot remains a useful concurrency example: [#11498](https://github.com/nielifeng/seatunnel/actions/runs/29799689096), [#11465](https://github.com/nielifeng/seatunnel/actions/runs/29799688469), [#11464](https://github.com/nielifeng/seatunnel/actions/runs/29799686644), and [#10461](https://github.com/nielifeng/seatunnel/actions/runs/29795346775) expanded to 330 jobs. Exactly 20 were in progress while 169 remained queued. #11465 and #11464 had already failed a unit-test lane while another 109 jobs across those two runs were still running or queued.

The larger sample confirms that long attempts are not specific to one fork and are not artifacts of rerun gaps:

- [#11440](https://github.com/apache/seatunnel/pull/11440) ([run](https://github.com/nzw921rx/seatunnel/actions/runs/29413938915)): attempt 1, 83 expanded / 75 runnable jobs, failed after about 985 minutes.
- [#10973](https://github.com/apache/seatunnel/pull/10973) ([run](https://github.com/davidzollo/seatunnel/actions/runs/29235539989)): attempt 1, 83 expanded / 75 runnable jobs, failed after about 895 minutes.
- [#11190](https://github.com/apache/seatunnel/pull/11190) ([run](https://github.com/DanielLeens/seatunnel/actions/runs/28302698089)): attempt 1, 65 expanded / 36 runnable jobs, succeeded after about 743 minutes.
- [#11140](https://github.com/apache/seatunnel/pull/11140) ([run](https://github.com/fallintoplace/seatunnel/actions/runs/28919422490)): attempt 1, 83 expanded / 75 runnable jobs, failed after about 736 minutes.

The three most represented forks contributed 55% of valid sampled PRs, so PR-weighted statistics can still be influenced by prolific contributors. As a cross-check, among 33 forks with a timed completed attempt, the median of each fork's median duration was 138 minutes and the p90 of fork medians was 307 minutes. The long-feedback problem remains after reducing the weight of high-volume forks.

### Cost differs sharply by change scope

The table below uses first attempts and is descriptive rather than a stable per-tier SLO: engine, workflow, and API strata still contain far fewer than 30 timed observations. It nevertheless shows that broad risk flags erase the fast path for engine/API/workflow changes.

| Classified scope | First attempts / timed | Median expanded / runnable | First-attempt p50 / p90 |
| --- | ---: | ---: | ---: |
| Documentation-only | 56 / 56 | 49 / 7 | 75 / 101 min |
| Connector | 26 / 24 | 64 / 34 | 168 / 315 min |
| Zeta/engine | 3 / 3 | 81 / 69 | insufficient sample; max 895 min |
| API/cross-cutting | 10 / 7 | 83 / 75 | 447 / 620 min |
| Workflow | 5 / 3 | 83 / 75 | insufficient sample; max 985 min |

Among 15 timed first attempts with at least 80 expanded jobs, two succeeded and 13 failed. Their p50 was 454 minutes and p90 was 736 minutes. This is not a CI flakiness rate because failures may be genuine PR defects, and the sample is small; it measures how expensive it is to reach the first result with the current full matrix.

Across all 113 timed completed first attempts, successful attempts had p50/p90 of 79/285 minutes, while failed attempts had p50/p90 of 317/736 minutes. Scope and defect type are confounders, but the data contradicts the desired fail-fast behavior: failures are generally discovered later, not earlier.

### Job-level cost and critical path

The latest visible job records from 183 runs contained 11,558 expanded jobs. Summing only valid non-negative job intervals produced a lower bound of 371,683 runner-minutes (about 6,195 runner-hours); 120 non-skipped intervals were invalid or incomplete and were excluded. Because rerun views can combine jobs from different attempts, this is a workload-ranking signal rather than a billing total.

The top ten job families account for about 44% of the valid observed runtime:

| Job family | Valid runtime | Runtime p50 / p90 | Start-delay p90 | Completion-offset p90 |
| --- | ---: | ---: | ---: | ---: |
| Kafka connector IT | 25,116 min | 148 / 167 min | 383 min | 490 min |
| Unit tests | 21,690 min | 55 / 63 min | 142 min | 196 min |
| All connectors part 4 | 20,845 min | 146 / 234 min | 244 min | 490 min |
| Redis connector IT | 16,589 min | 133 / 143 min | 364 min | 497 min |
| All connectors part 7 | 16,525 min | 99 / 170 min | 213 min | 405 min |
| Transform part 2 | 13,283 min | 82 / 133 min | 330 min | 490 min |
| All connectors part 3 | 12,699 min | 84 / 131 min | 248 min | 409 min |
| File-local connector IT | 12,369 min | 89 / 99 min | 355 min | 490 min |
| JDBC part 1 | 12,231 min | 68 / 106 min | 177 min | 401 min |
| All connectors part 1 | 11,705 min | 67 / 113 min | 180 min | 375 min |

`Start delay` includes both dependency waiting and runner queueing, so it cannot attribute all delay to the GitHub runner scheduler. It does show that many critical-path suites begin three to six hours after the workflow attempt starts. The problem is therefore both execution cost and scheduling/fan-out, not either one alone.

One successful first-attempt full-matrix run, [#11220](https://github.com/apache/seatunnel/pull/11220) ([run](https://github.com/zhangshenghang/seatunnel/actions/runs/28459284750)), provides a useful step-level breakdown:

- 62 `free disk space` steps consumed about 76 runner-minutes in total (p50 about one minute, p90 about two minutes).
- The combined website install/build step took about 74 minutes.
- Checkout and Java setup were small relative to the Maven integration-test steps.
- Individual Maven E2E steps ranged from minutes to more than 200 minutes, but currently combine reactor build and test execution, so CI cannot yet quantify how much is duplicate compilation versus the actual test.

This changes the optimization priority: disk cleanup and action-version upgrades are worthwhile hygiene, but routing, gating, reactor reuse, and the top E2E families have materially larger leverage.

### One compile defect is amplified into many failing lanes

[PR #10306](https://github.com/apache/seatunnel/pull/10306) changed API/core behavior and produced [84 expanded / 75 runnable jobs](https://github.com/zhangshenghang/seatunnel/actions/runs/29685279927). Its latest attempt failed in about 20 minutes, but still produced 25 failed and 25 cancelled lanes. Logs from independent JDBC partitions, including [`jdbc-connectors-it-part-2`](https://github.com/zhangshenghang/seatunnel/actions/runs/29685279927/job/88237807599) and [`jdbc-connectors-it-part-4`](https://github.com/zhangshenghang/seatunnel/actions/runs/29685279927/job/88237807606), show the same `connector-jdbc` compilation defects: missing Lombok-generated `log`, missing `getJdbcConnectionConfig()`, missing builder/getter methods, and constructor mismatch.

The E2E test itself was not needed to find this defect. Multiple lanes independently restored an approximately 995 MB Maven cache and rebuilt the same reactor before failing on the same compile errors. A full run has approximately 40 `setup-java` Maven-cache consumers, so cache restore traffic alone can reach tens of GB per build before counting repeated compilation and packaging. The correct optimization order is therefore:

1. compile/unit gate once before E2E fan-out;
2. stop global fan-out after a required gate failure;
3. build immutable artifacts once and reuse them in test partitions.

This example is useful precisely because it failed quickly: even when the root compilation defect is immediately reproducible, 75 runnable jobs are created and many independent lanes repeat the same failure instead of being held behind one compile gate.

### Workflow structure explains the observations

- `backend.yml` has 47 top-level job definitions, 53 Maven invocations, 38 `free_disk_space.sh` invocations, and about 40 jobs restoring a Maven cache via `setup-java`.
- Forty-two jobs depend directly on only `changes` and `sanity-check`; 29 jobs use the broad `api || engine` routing condition, and 38 Maven E2E commands rebuild dependencies with `-am`.
- The unit matrix runs JDK 8/11 on Ubuntu/Windows and invokes the full reactor `clean verify`.
- The `changes` job computes `ut-modules`, but the unit-test command does not use that output: a narrow non-API change still executes full-reactor `clean verify` in all four OS/JDK lanes.
- `document` runs for every `api=true` change, `seatunnel-ui` also runs for every `api=true` change, and dependency-license checks run for every `api || engine` change. These auxiliary checks are routed by the same coarse risk flag rather than docs/UI/dependency-file-specific flags.
- `engine=true` enables engine, transform, all connector partitions, all JDBC partitions, and specialized connector E2E jobs. A narrow logging, metrics, or storage fix can therefore receive nearly the same fan-out as a cross-cutting API change.
- Expensive jobs depend on `changes` and `sanity-check`, not on a successful changed-module compile/unit gate.
- Of 41 recent Build runs in the originally sampled `nielifeng` fork, 23 were cancelled after newer pushes. This is a run-count observation, not runner-minute waste; work completed before cancellation is nevertheless discarded.

PR #9976 contains useful work that should be integrated rather than duplicated: duration analysis, disk-cleanup instrumentation, slow-module extraction, and connector-shard rebalancing are supported by the job-level results above. Its current 20-second delayed-scheduling design prioritizes long E2E jobs before unit tests and other short jobs. Longest-processing-time-first can reduce a single workflow's makespan when capacity is fixed, but before a compile/unit gate it harms time-to-first-failure and can let one PR occupy a fork's limited concurrency. Long-job prioritization should therefore run only after the fast gate and inside a bounded dynamic E2E matrix.

### Status and trust evidence

Thirty-four of 185 valid historical samples had a mismatch between the upstream `Build` check and actual fork run at collection time. Because the sample includes closed PRs, this is evidence that checks can remain permanently unconverged, not a direct estimate of the live-open-PR mismatch rate.

The live failure mode was separately observed on three open PRs whose `mergeable_state` was `blocked`:

- [#11198](https://github.com/apache/seatunnel/pull/11198) ([run](https://github.com/srijan-singh/seatunnel/actions/runs/29805736323)): parent run and upstream check reported `queued`, while 20 child jobs were in progress and 38 were queued.
- [#11414](https://github.com/apache/seatunnel/pull/11414) ([run](https://github.com/ss666/seatunnel/actions/runs/29804632149)): parent run and upstream check reported `queued`, while 20 child jobs were in progress and 36 were queued.
- [#11503](https://github.com/apache/seatunnel/pull/11503) ([run](https://github.com/nielifeng/seatunnel/actions/runs/29795346775)): parent run and upstream check reported `queued`, while six child jobs were in progress.

The implementation makes these inconsistencies possible: `notify_test_workflow.yml` creates `Build` as `queued`; `update_build_status.yml` polls only every 15 minutes, excludes `mergeable_state=blocked`, and copies the reusable-workflow caller status instead of deriving activity from its child jobs.

The actual build logs also identify the called workflow as the contributor branch, for example `zhangshenghang/seatunnel/.github/workflows/backend.yml@refs/heads/feat-add-error-handle`, while `build_main.yml` grants `packages: write`. This does not by itself demonstrate an exploit, but it does show that a required upstream check currently trusts fork-controlled workflow logic with broader permission than a no-secret PR validation path should need.

Relevant workflow locations:

- Broad unit matrix and full reactor verify: https://github.com/apache/seatunnel/blob/dev/.github/workflows/backend.yml#L380-L400
- Engine changes triggering broad connector/JDBC E2E: https://github.com/apache/seatunnel/blob/dev/.github/workflows/backend.yml#L648-L669 and https://github.com/apache/seatunnel/blob/dev/.github/workflows/backend.yml#L794-L821
- Fork Build on every branch push: https://github.com/apache/seatunnel/blob/dev/.github/workflows/build_main.yml#L20-L32
- Synthetic Build check initialized as queued: https://github.com/apache/seatunnel/blob/dev/.github/workflows/notify_test_workflow.yml#L75-L154
- Scheduled status synchronization and mergeable-state filter: https://github.com/apache/seatunnel/blob/dev/.github/workflows/update_build_status.yml#L46-L100

## Root causes

### 1. Path classification is too coarse

`seatunnel-engine/**` is effectively treated as one risk domain. That is conservative, but it makes observability, logging, storage, runtime, API-contract, and test-only changes trigger similar E2E coverage.

### 2. Static job fan-out is larger than available concurrency

Each engine PR creates approximately 69 runnable jobs. Several concurrent PRs share the contributor account's runner quota, so the critical path becomes queue time rather than test execution time. There is no per-workflow fairness control.

### 3. Expensive jobs start before a meaningful failure gate

Most integration jobs depend on `changes` and `sanity-check`, but not on successful changed-module unit tests. A unit failure therefore does not stop unrelated expensive work.

### 4. Every E2E job rebuilds much of the reactor

Jobs repeatedly checkout the repository, restore Maven state, free disk space, and invoke Maven with `-am`. Caching helps dependency downloads but does not eliminate repeated reactor compilation and packaging.

### 5. Status reporting is indirect and can be stale

The required check is created in the upstream repository while execution happens in a fork. A scheduled workflow polls the fork and mirrors the top-level status. This adds delay and can report `queued` even while reusable-workflow child jobs are active.

### 6. Required-check workflow integrity depends on fork-controlled definitions

The Build workflow executed in a contributor fork comes from the contributor branch, and the upstream check mirrors that result. The desired trust model should use a fixed upstream workflow definition for required checks, with read-only permissions for untrusted PR code. Secret-bearing tests should run only after maintainer approval or in a protected merge-queue context.

## Proposed target architecture

### Tier 0: fast gate (required for every PR)

- license headers, formatting, workflow validation;
- changed-module compilation and unit tests on Linux;
- impact/risk classification output;
- target: first actionable result within 10 minutes.

### Tier 1: impacted validation (required for normal PRs)

- module dependency graph determines affected unit and integration modules;
- split engine scope into API/contract, runtime, server, storage, observability, UI, and test-only categories;
- run Linux JDK 8/11 for affected modules;
- run a small representative engine/connector canary set for medium-risk engine changes;
- target: required checks complete within 30-60 minutes for normal PRs.

### Tier 2: exhaustive validation

- full connector/JDBC partitions;
- full JDK/OS matrix, including Windows compatibility;
- secret-dependent and long-running E2E;
- run for merge-queue candidates, explicit `full-ci` requests, cross-cutting API/workflow changes, and nightly builds.

### Build once, test many

- build the distribution/relevant reactor artifacts once per SHA and JDK;
- upload immutable artifacts identified by commit SHA, JDK, Scala version, and Maven profile;
- E2E jobs download those artifacts instead of rebuilding with `-am`;
- retain isolated test execution while removing repeated compilation.

### Trusted execution and direct reporting

- run standard PR checks from an upstream `pull_request` workflow with read-only permissions and no secrets;
- use maintainer approval/labels or merge queue for secret-bearing workflows;
- report required checks directly from the executing workflow;
- if cross-repository mirroring must remain temporarily, synchronize blocked PRs and derive status from child jobs instead of only the reusable-workflow caller status.

## Task list

### Phase 0: measurement and status correctness

- [ ] Define CI SLO metrics: queue time, first-result time, required-check duration, runner minutes, cancellation waste, and flaky retry rate.
- [ ] Add a machine-readable per-run summary with selected risk tier, expanded jobs, runnable jobs, skipped jobs, queue duration, and longest critical-path jobs.
- [ ] Measure cache bytes restored/saved, artifact transfer bytes, and repeated compiler/reactor work per run; do not infer runner minutes from invalid job timestamps.
- [ ] Fix Build check synchronization for `blocked` PRs.
- [ ] Report `in_progress` when child jobs are active, even if the reusable-workflow caller remains queued.
- [ ] Replace or supplement the 15-minute polling workflow with an event/callback-based update.
- [ ] Add status-mirroring regression tests for queued, child-active, completed, cancelled, rerun, missing-run, and blocked-PR states.

### Phase 1: fast gate and failure propagation

- [ ] Add a compile-once plus changed-module unit-test gate before expensive E2E fan-out, and publish its immutable outputs for downstream jobs.
- [ ] Use the existing `ut-modules` output for non-API unit tests instead of always running full-reactor `clean verify` in all four OS/JDK lanes.
- [ ] Split docs, UI, dependency-file, workflow-control-plane, API-contract, and engine flags so unrelated auxiliary/full-matrix jobs are not selected through one `api` flag.
- [ ] Cancel queued/running expensive jobs when a required fast gate fails.
- [ ] Add workflow validation tests for change-routing rules and generated matrices.
- [ ] Add per-workflow `max-parallel` limits or another fairness policy so one PR cannot starve all other builds.

### Phase 2: impact-based dynamic matrices

- [ ] Split the current `engine` flag into risk-relevant engine scopes.
- [ ] Extend the existing module dependency analysis instead of maintaining a second dependency model.
- [ ] Generate unit/E2E matrices dynamically from the changed modules and risk tier.
- [ ] Define a documented connector canary set for engine runtime changes.
- [ ] Preserve a maintainer-controlled `full-ci` override for uncertain impact classifications.
- [ ] Integrate the slow-module extraction, rebalancing, and measurement work in #9976, but apply long-job prioritization only after the fast gate.
- [ ] Resolve dependency pollution tracked by #11074 and add regression tests for routing output.

### Phase 3: matrix and artifact optimization

- [ ] Keep Linux JDK 8/11 as the normal PR compatibility matrix.
- [ ] Move full Windows and exhaustive connector/JDBC matrices to Tier 2, retaining a small Windows compile/smoke check where appropriate.
- [ ] Build immutable test artifacts once per SHA/JDK/profile and reuse them across E2E jobs.
- [ ] Pilot artifact reuse first on Kafka, Redis, all-connectors part 4, and one JDBC partition, then compare reactor-build time, artifact transfer, and test time separately.
- [ ] Remove unnecessary repeated reactor builds and limit disk cleanup to jobs that need it; verify downstream E2E jobs do not recompile the shared reactor.
- [ ] Measure cache hit rate and transferred bytes, and avoid concurrent cache-save contention.

### Phase 4: trusted upstream execution

- [ ] Move required no-secret checks to a fixed upstream `pull_request` workflow with least-privilege permissions.
- [ ] Define the approval and SHA-pinning model for secret-bearing or privileged E2E.
- [ ] Remove fork workflow discovery/status mirroring after the upstream path is stable.
- [ ] Add merge-queue full validation and retain nightly exhaustive validation.

## Acceptance criteria

- The PR page reflects queued/running/completed state within five minutes, including PRs blocked on review or other checks.
- A normal narrow engine PR creates no more than 15 runnable jobs unless explicitly promoted to full CI.
- Fast-gate feedback is available within 10 minutes under normal runner availability.
- Fast-gate jobs have p90 start delay below 10 minutes and are never intentionally delayed behind long E2E jobs.
- Normal required PR checks target p50 <= 45 minutes and p90 <= 90 minutes, reported separately for docs, narrow connector, engine-internal, and API/cross-cutting risk tiers over at least 30 runs per tier.
- A required fast-gate failure stops new expensive jobs and cancels outstanding expensive jobs within five minutes.
- Failed first attempts target p90 <= 30 minutes for compile/unit defects, compared with the observed 736-minute p90 across all first-attempt failures.
- A compile failure is reported by the fast gate and does not recur independently across E2E partitions.
- Downstream E2E partitions consume SHA/JDK/profile-pinned artifacts and do not rebuild the shared reactor; CI reports cache/artifact transfer bytes so the reduction is auditable.
- The top ten job families and total valid runner-minutes are reduced by at least 40% for normal PR validation, with exhaustive coverage preserved in Tier 2.
- Exhaustive JDK/OS/connector coverage remains required for high-risk changes or merge-queue candidates and runs nightly.
- Required-check workflow logic comes from a trusted upstream definition; untrusted PR code executes with read-only permissions and no secrets.
- CI routing has automated tests covering representative docs, connector, engine-storage, engine-server, engine-contract, API, and workflow changes.
- Status reporting has automated tests proving that upstream checks converge correctly for child-active, blocked, cancelled, rerun, and completed fork workflows.

## Risks and mitigations

- **False-negative impact classification:** keep connector canaries, merge-queue/full nightly validation, routing regression tests, and a `full-ci` override.
- **Artifact mismatch:** key artifacts by SHA/JDK/Scala/profile and verify metadata before tests.
- **Reduced Windows coverage on ordinary PRs:** retain Windows smoke checks and run the full Windows matrix for Tier 2 and nightly validation.
- **Self-hosted runner operational/security cost:** optimize selection and duplicate work before treating runner expansion as the primary solution.

## Non-goals

- Removing full E2E coverage.
- Treating additional runner capacity as the only fix.
- Running untrusted PR code with repository secrets or write permissions.
- Replacing existing incremental-test work that can be extended safely.

Start by reviewing the existing GitHub Actions workflow definitions and the Build check's expanded job structure, including setup-java, Maven, and E2E lanes. Compare the current fan-out and failure propagation with the proposed risk tiers, artifact reuse, and direct reporting. Done means the CI architecture has an agreed implementation plan and measurable fast-feedback behavior.

Written by the indexing model from the issue text.