[Umbrella] Delta-on-Gluten: follow-ups after the Delta Spark UT pipeline

The Delta Spark UT pipeline landed in #12388 (fixes #9296). It runs delta-io/delta's `spark` test suite against a Gluten Velox bundle and gates each run against a committed baseline of known failures.

That baseline is currently **735 known failures out of 19,073 tests**. This issue tracks the work to bring that number down and to improve the pipeline itself. It is a tracking issue only — discussion belongs in the linked issues.

## 1. Correctness gaps behind the 735 baseline entries

These are not a consequence of Gluten declining to offload: a plan Gluten cannot offload should fall back to vanilla Spark and the test should still pass. Most entries below are bugs -- fallback not happening, metrics differing from vanilla, or the native side failing. A separate group is expected rather than fixable here: tests that assert on the query plan see a different plan once Gluten offloads, which is by design. Those are worth splitting out during triage, since they should either be fixed upstream (as delta-io/delta#7104 and delta-io/delta#7105 were) or stay in the baseline permanently.

The largest clusters, counted from `known-failures.txt` by suite:

| Tests | Area | Issue |
|---:|---|---|
| 130 | `DescribeDeltaHistorySuite` + `...WithCatalogOwnedBatch100Suite` | needs triage |
| 100 | `stats.DataSkippingDeltaV1*` (7 suite variants, 34 of them column-mapping) | needs triage |
| 64 | `ImplicitMergeCastingSuite` + `ImplicitStreamingMergeCastingSuite` (overflow / storeAssignmentPolicy) | needs triage |
| 42 | `commands.backfill.RowTrackingBackfillConflicts(DV)Suite` | needs triage |
| 14 | `io.delta.tables.DeltaTableHadoopOptionsSuite` — Hadoop FS options | #4963 |

Two clusters cut across suites, so they don't show up in the table above (counts from the triage in #12388): **~226** tests failing on `IncrementMetric` (#9003) and **~47** on the Delta stats tracker `ClassCastException` when the stats plan can't be offloaded (#12538).

Related type-support gap: TIMESTAMP_NTZ (#11622), which is one trigger of #12538.

- [ ] #9003 — IncrementMetric
- [ ] #12538 — Delta stats tracker should fall back instead of throwing
- [ ] #11622 — TIMESTAMP_NTZ support
- [ ] #4963 — Hadoop file system options
- [ ] Triage the `DescribeDeltaHistory`, `ImplicitMergeCasting`, `RowTrackingBackfill` and `DataSkipping` clusters and file one issue per root cause

## 2. Native bugs currently worked around in CI

Both of these are *hidden* by the pipeline today, so the workarounds should be removed together with the fixes.

- [ ] #12377 — DV bitmap aggregator aborts on an invalid row index. Currently quarantined by error signature in `flaky-error-patterns.txt` (2 patterns). Remove those entries when fixed.
- [ ] #12387 — native memory OOM when reading billions-of-rows tables. Two `DeletionVectorsSuite` 2B-row tests are force-failed in `setup-delta.sh` to stop them OOM-killing the shard. Remove that patch when fixed. PR: #12866
- [ ] Unsupported operations should induce a **fallback** for Delta plans rather than failing (raised by @zhztheplayer in #12388; #12538 is the concrete instance)

## 3. Pipeline improvements

- ~~**Automatically remove now-passing tests from the Delta baseline.** #12855 opens or updates a deletion-only PR from the scheduled run, covering core-library fixes such as #12815 and #12851 that did not trigger Delta CI when merged.~~. Setting to allow Github Actions to create PR is not enabled
- [x] **Move compiler and dependency caches to Apache Stash.** Migrated ccache, Maven, and sbt/Ivy/Coursier caches in #12820.
- [ ] **Cache and reuse the fully built Velox native library across workflows (not ccache).** `gluten-delta/**` and `backends-velox/src-delta*/**` match both `velox_backend_x86.yml` and `delta_spark_ut.yml`, so those PRs build the native lib twice (~10 min). Suggested by @philo-he and @zhouyuan. A prebuilt-library cache is preferable to artifact-passing: artifacts are scoped to a workflow *run* and there is no cross-workflow `needs`, so the Delta workflow would have to poll a sibling run it races with, whereas a prebuilt-library cache populated by push-to-`main` has no ordering problem. **The key must cover `cpp/**`, the build flags and the base image** — unlike ccache, where a bad key only costs a rebuild, a stale hit on a prebuilt lib means silently testing the wrong binary. A miss must fall back to building.
- [ ] **Cover more Spark versions.** The workflow already takes `spark_version` / `delta_ref` inputs, and `pom.xml` pairs Spark 3.5→Delta 3.3.2, 4.0→4.0.1, 4.1→4.1.0. The blocker is the baseline: each combination needs its own `known-failures.txt` and a bootstrap run. Suggested starting point: **Spark 3.5 as a nightly-only second leg**, which keeps per-PR cost unchanged. Requested by @zhouyuan.
- [ ] **Remove the overlapping imported Delta suites.** @malinjawi identified 8 suites imported from delta-io/delta (`DeltaSuite`, `DeleteSQLSuite` + DV variants, `UpdateSQLSuite`, `DeltaDDLSuite`, `DeltaInsertIntoTableSuite`, `ClusteredTableClusteringSuite`, `OptimizeMetadataOnlyDeltaQuerySuite`, `OptimizedWritesSuite`) that this pipeline now duplicates — but **only on the Spark 4.1 leg**. They are the only Delta coverage on Spark 3.5 and 4.0, so this is blocked on the Spark 3.5 leg above. Raised by @zhztheplayer.
- [ ] **Delta 4.3.0 support.** Attempted during #12388: the bundle is compiled against Delta 4.1.0 and hits a binary-incompatible change (`IdentityColumn.logTableWrite`, first param `Snapshot` → `SnapshotDescriptor`), which `NoSuchMethodError`s on every write. Needs the bundle built against 4.3.0.
- [X] **`/delta-test` PR comment opt-in.** Replaces the proposed `run-delta-ci` label: contributors cannot apply labels, but anyone can comment `/delta-test` to force the full suite on a PR skipped by the `paths:` filter. Implemented in #12781.
- [ ] **Drop the upstream cherry-picks** (delta-io/delta#7104 + #7105, `FileSourceScanExec` → `FileSourceScanLike`) once `DELTA_REF` moves to a release that contains them. `setup-delta.sh` already skips a cherry-pick that is already applied, so this is cleanup.

## Notes for whoever picks these up

- Refresh the baseline by running **Delta Spark UT (Gluten)** with `update_baseline=true` and committing the produced artifact; see `.github/workflows/util/delta-spark-ut/README.md`. - A test that fails **consistently** belongs in `known-failures.txt`. `flaky-tests.txt` / `flaky-error-patterns.txt` are only for tests that genuinely pass on some runs and fail on others — quarantining a deterministic failure hides a real bug. - The nightly run enforces `fail_on_fixed=true`, so when a fix lands, the nightly goes red until the now-passing tests are removed from the baseline. That is the intended signal.

This is a tracking issue, so start by selecting one unchecked follow-up and read the referenced issue. For pipeline work, begin with .github/workflows/util/delta-spark-ut/README.md, known-failures.txt, and setup-delta.sh; done means the selected change is implemented, validated by the Delta Spark UT workflow, and the baseline or workaround is updated as appropriate.

Written by the indexing model from the issue text.