apache / apache/texera

Generalize the amber-integration test setup into a shared cross-module integration CI job

Open
#6,047 0 comments 0 reactions 1 assignee Claimed by @mengw15 View on GitHub
Dominant language
Scala
Stars
314
Forks
187
Avg merge
1d 21h
Merged PRs (30d)
214

Description

### Task Summary

Integration-test support is currently hard-wired to the `amber` module only. Turn it into a **reusable mechanism any module can opt into** — a per-module integration source set that compiles and runs on the module's *own* classpath, with a shared `@IntegrationTest` tag and shared sbt settings — so integration specs live next to the code they exercise, in whatever module that is.

> **Re-scoped (2026-07):** the original proposal also called for collapsing everything into a *single* cross-module `integration` job. That direction is superseded — CI now runs **per-tier** integration jobs (`amber-integration`, and `platform-integration` from #6274) whose infra provisioning is being shared via a composite action (#6275). So this issue is now just the **sbt-side mechanism + rehoming the stranded specs**: a module's integration specs run in the existing job for its tier, on the module's own classpath. No unified job.

**Trigger (from #6045):** the shared `VirtualDocumentSpec` base and three live-catalog iceberg specs exercise `common/workflow-core` code, so their specs ideally live there — but common has no integration-test capability today, and pulling common's test scope into amber (`WorkflowCore % "test->test"`) causes a classpath conflict (Hadoop → `com.sun.jersey` 1.19's JAX-RS 1.x `Response` shadows amber's `javax.ws.rs-api:2.1.1`, breaking `HuggingFaceModelResourceSpec`). #6045 left those specs stranded in `amber` as a stopgap.

common/workflow-core is just the **first module to need this** — the same capability should be available to the other Scala modules (`*-service`, `WorkflowOperator`, `DAO`, `Auth`, …) as they grow integration coverage.

### Current state

- The `@IntegrationTest` tag lives at `amber/src/test/integration/.../amber/tags/IntegrationTest.java` — in amber's package and source dir.
- The wiring (`Test / unmanagedSourceDirectories += src/test/integration` + the `AMBER_TEST_FILTER` skip/only switch) is only in `amber/build.sbt`.
- Only `amber` has a `src/test/integration` source set.
- The `amber-integration` CI job runs only `WorkflowExecutionService/test` (integration-only) + amber Python integration tests — no other module's integration specs. (`platform-integration` (#6274) exists for the platform tier but hosts boot smoke tests, not tagged integration specs yet.)

### Suggested direction

Reverse the coupling: instead of dragging a module's test scope into `amber` (which is what causes the classpath conflict), give each module its **own** integration source set running on its **own** classpath.

- Move the `@IntegrationTest` tag to a shared location (e.g. `common`) so any module can reference it.
- Factor the integration source-set + `AMBER_TEST_FILTER` wiring into **shared, reusable sbt settings** (an AutoPlugin or a shared settings sequence) applied to each participating module — not copy-pasted per `build.sbt`.
- Run each module's integration-tagged specs on that module's **own classpath**, inside the existing per-tier integration job — e.g. common's specs in `amber-integration` as `WorkflowCore//test`, alongside amber's, rather than in a single unified job.
- First adopter: move the iceberg specs + `VirtualDocumentSpec` from `amber` back into `common/workflow-core/src/test/integration`. Other modules opt in incrementally as they add integration coverage.

Why this avoids the #6045 conflict: each module's integration specs compile and run on that module's own classpath (common already has Hadoop as a legit main dep), so common's `jersey` 1.19 is never mixed into amber's `javax.ws.rs` 2.1 classpath.

Follow-up to #6045; related to #6034, #6274, #6275.

### Task Type

- [x] Refactor / Cleanup
- [x] DevOps / Deployment / CI
- [x] Testing / QA
- [ ] Documentation
- [ ] Performance
- [ ] Other

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.