apache / apache/texera

ci: deduplicate Scala compilation across CI jobs via a prebuild + artifact handoff

Open
#6,318 1 comment 0 reactions 1 assignee Claimed by @Yicong-Huang View on GitHub
Dominant language
Scala
Stars
314
Forks
187
Avg merge
2d 8h
Merged PRs (30d)
209

Description

### Task Summary

The four sbt-based CI jobs — `amber`, `amber-integration`, `platform`, and `platform-integration` — each compile the shared common-module graph (DAO, WorkflowCore, WorkflowOperator, Auth, Config, …) from scratch. Across their matrices that is up to **14 runner legs** redoing the same Scala build on every run, wasting runner-minutes and risking build inconsistency.

Proposal: add a single `scala-build` **prebuild** job that compiles the whole Scala world once (main + test scopes), runs the cross-cutting scalafmt/scalafix lints, and builds all dists, then uploads the compiled `target/` tree and the dist zips as artifacts. The downstream Scala jobs download and reuse them instead of recompiling — `amber` / `amber-integration` (Linux) / `platform` restore the compiled state for their `jacoco`/`test`, and `platform-integration` pulls just the dists for smoke-boot.

Design constraints:
- Respect the existing label gate: the prebuild runs only when at least one Scala stack is requested (union of `run_amber` / `run_amber_integration` / `run_platform` / `run_platform_integration`), so docs-only / frontend-only PRs never trigger it.
- macOS `amber-integration` keeps self-compiling — Zinc analysis embeds OS-specific absolute paths and is not portable across runners.
- Primary goal is **deduplicating compilation / cutting runner-minutes**; wall-clock is expected to stay roughly flat and will be measured against a baseline before committing to the approach.

### Task Type

- [ ] Refactor / Cleanup
- [x] DevOps / Deployment / CI
- [ ] Testing / QA
- [ ] Documentation
- [ ] Performance
- [ ] Other

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.