apache / apache/texera

Remove the unused hadoop Iceberg catalog type

Open
#6,150 1 comment 0 reactions 1 assignee Claimed by @anthonychengit View on GitHub
Dominant language
Scala
Stars
314
Forks
187
Avg merge
1d 21h
Merged PRs (30d)
214

Description

### Task Summary

`storage.iceberg.catalog.type` accepts `hadoop`, `rest`, or `postgres`. Production uses `rest` (Lakekeeper); `postgres` is the local/CI backend. The `hadoop` type — an Iceberg `HadoopCatalog` — appears unused and is worth dropping:

- **Not configured / not selected anywhere.** `storage.conf` has `rest { … }` and `postgres { … }` blocks but no `hadoop { … }` block; `hadoop` only appears in the `# either hadoop, rest, or postgres` comment. No deployment (prod / local-dev / single-node) selects it.
- **Only safe on HDFS, which we don't use.** The [Iceberg docs](https://iceberg.apache.org/docs/latest/java-api-quickstart/) state a Hadoop catalog "can only be used with HDFS or similar file systems that support atomic rename." Texera stores Iceberg data on S3/MinIO and never on HDFS — exactly the object-store case that lacks atomic rename — so `hadoop` was never a viable backend here. (Not formally `@Deprecated` in Iceberg 1.7.1, just documented as unsuitable.)
- **Confirmed in practice.** Evaluating `hadoop` as the unit-job catalog in #6049 flooded `CommitFailedException: Version N already exists` / `Unable to acquire lock` under amber's concurrent result-port commits.
- **Less to maintain** — one fewer catalog backend.

**Seeking input:** does anyone rely on the `hadoop` catalog type? If not, we'll remove it.

### Scope

- `IcebergCatalogInstance`: drop the `case "hadoop" => IcebergUtil.createHadoopCatalog(…)` branch (the `case _` already rejects unsupported types).
- `IcebergUtil`: remove `createHadoopCatalog`; narrow the import `{HadoopCatalog, HadoopFileIO}` → `HadoopFileIO` (still used by the `rest`/`postgres` catalogs).
- `storage.conf`: update the comment to `# either rest or postgres`.
- No tests or docs reference the type (confirmed by grep).

Suggested by @Yicong-Huang in #6049.

### Task Type

- [x] Refactor / Cleanup
- [ ] DevOps / Deployment / CI
- [ ] Testing / QA
- [ ] Documentation
- [ ] Performance
- [ ] Other

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.