Default the Iceberg catalog to REST and keep the amber unit job free of a live catalog
- Dominant language
- Scala
- Stars
- 314
- Forks
- 187
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 214
Description
### Task Summary
Production uses the REST (Lakekeeper) Iceberg catalog, but `common/config/src/main/resources/storage.conf` still defaults `iceberg.catalog.type` to `postgres`. CI should stay close to production. Flip the default to `rest`, and first make sure the amber **unit** job no longer depends on a live Iceberg catalog so the flip doesn't break it. Scope is ~3 ordered tasks (Task 1 must land before Task 2, or the amber job breaks).
Context: prod = REST; local-dev and single-node already override to `rest` via env, so they're unaffected by the default. Separate from #6033 (which only removes the obsolete `bootstrap-lakekeeper.sh`).
**Task 1 — Move the live-catalog iceberg specs into integration.**
`IcebergDocumentSpec`, `IcebergDocumentConsoleMessagesSpec`, `IcebergTableStatsSpec`, `IcebergUtilSpec` (in `common/workflow-core/src/test/scala/.../storage/result/iceberg/` and `.../util/`) resolve a live catalog via `IcebergCatalogInstance.getInstance()`. Today the amber job runs them against the postgres catalog it provisions; after Task 2 they would try to reach Lakekeeper, which the unit job has no reason to run. Relocate them to `amber/src/test/integration/` (where `IcebergRestCatalogIntegrationSpec` already lives and the integration filter applies). Note: tagging in place is **not** sufficient — `common/workflow-core/build.sbt` does not read `AMBER_TEST_FILTER` (only `amber/build.sbt` does), so `@IntegrationTest` would be ignored there; relocating (or wiring the same filter into `common/workflow-core/build.sbt`) is required.
**Task 2 — Flip the default.**
Change `iceberg.catalog.type` in `storage.conf` from `postgres` to `rest`.
**Task 3 — Make CI integration mirror prod, keep both catalogs covered.**
Run `amber-integration` against the REST catalog so the integration suite matches production (Lakekeeper is already provisioned there). Keep postgres-catalog coverage in the integration suite as well — `postgres` stays a supported `iceberg.catalog.type`, so it should still be exercised somewhere, just not in the unit job. The only change to the amber **unit** job is that it stops needing a live Iceberg catalog for these specs; postgres catalog support itself is untouched.
### Task Type
- [x] Refactor / Cleanup
- [x] DevOps / Deployment / CI
- [x] Testing / QA
- [ ] Documentation
- [ ] Performance
- [ ] Other
Contributor guide
Assessment
This issue has not been assessed yet.