Remove the unused hadoop Iceberg catalog type
- Dominant language
- Scala
- Stars
- 314
- Forks
- 187
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 214
Description
### Task Summary
`storage.iceberg.catalog.type` accepts `hadoop`, `rest`, or `postgres`. Production uses `rest` (Lakekeeper); `postgres` is the local/CI backend. The `hadoop` type — an Iceberg `HadoopCatalog` — appears unused and is worth dropping:
- **Not configured / not selected anywhere.** `storage.conf` has `rest { … }` and `postgres { … }` blocks but no `hadoop { … }` block; `hadoop` only appears in the `# either hadoop, rest, or postgres` comment. No deployment (prod / local-dev / single-node) selects it.
- **Only safe on HDFS, which we don't use.** The [Iceberg docs](https://iceberg.apache.org/docs/latest/java-api-quickstart/) state a Hadoop catalog "can only be used with HDFS or similar file systems that support atomic rename." Texera stores Iceberg data on S3/MinIO and never on HDFS — exactly the object-store case that lacks atomic rename — so `hadoop` was never a viable backend here. (Not formally `@Deprecated` in Iceberg 1.7.1, just documented as unsuitable.)
- **Confirmed in practice.** Evaluating `hadoop` as the unit-job catalog in #6049 flooded `CommitFailedException: Version N already exists` / `Unable to acquire lock` under amber's concurrent result-port commits.
- **Less to maintain** — one fewer catalog backend.
**Seeking input:** does anyone rely on the `hadoop` catalog type? If not, we'll remove it.
### Scope
- `IcebergCatalogInstance`: drop the `case "hadoop" => IcebergUtil.createHadoopCatalog(…)` branch (the `case _` already rejects unsupported types).
- `IcebergUtil`: remove `createHadoopCatalog`; narrow the import `{HadoopCatalog, HadoopFileIO}` → `HadoopFileIO` (still used by the `rest`/`postgres` catalogs).
- `storage.conf`: update the comment to `# either rest or postgres`.
- No tests or docs reference the type (confirmed by grep).
Suggested by @Yicong-Huang in #6049.
### Task Type
- [x] Refactor / Cleanup
- [ ] DevOps / Deployment / CI
- [ ] Testing / QA
- [ ] Documentation
- [ ] Performance
- [ ] Other
Contributor guide
Assessment
This issue has not been assessed yet.