Feature: HDFS commits for optimizing/maintenance as table owner (Hadoop proxy impersonation)
- Dominant language
- Java
- Stars
- 1.2k
- Forks
- 395
- Avg merge
- 4d 10h
- Merged PRs (30d)
- 33
Description
### Summary
On secured Hadoop clusters, AMS and optimizers typically run as a **dedicated service principal** (e.g. `amoro`). Iceberg/Hive warehouse paths are often owned by the **table creator** (e.g. `lily`). Optimizing **tasks** can succeed while the **AMS commit** fails with HDFS `AccessControlException` on `data/` or `metadata/` because all catalog file IO runs as the service user, not the table owner.
Configuring `hadoop.proxyuser.amoro.*` on the cluster is **necessary but not sufficient**: nothing in the AMS optimizing commit path uses Hadoop proxy impersonation today.
### Environment (example)
- Amoro 0.9.x (AMS + external optimizer)
- Kerberos, Hive Metastore + Iceberg tables on HDFS
- Ranger HDFS/Hive policies for the service user
- Tables under e.g. `/warehouse/tablespace/external/hive///` with owner = end user, mode `drwxr-xr-x`
### Observed behavior
1. Optimizer finishes MINOR (or similar) work; UI shows tasks **1/1 success**.
2. AMS commit fails; optimizing process status **FAILED**.
3. AMS logs show `OptimizingCommitException` / Iceberg commit failure, root cause similar to:
`Permission denied: user=amoro, access=WRITE, inode=".../mor_events/metadata":lily:hive:drwxr-xr-x`
Ranger grants for `amoro` on `/*` and cluster `hadoop.proxyuser.amoro.{hosts,groups,users}=*` did **not** change this, because commits still execute as `amoro`.
Workarounds that work operationally but do not scale well:
- Per-table or per-DB HDFS ACLs for the service user on `data/` and `metadata/`
- Warehouse-wide default ACLs
- Treating the service user as a broad HDFS/Ranger “power user”
These are painful in multi-tenant warehouses where table ownership follows the creating user (as with Hive).
### Expected behavior
When a catalog property (or table property) enables **owner impersonation**, AMS and optimizers should perform HDFS reads/writes for optimizing commits (and related maintenance) **as the table owner** (from HMS/Iceberg metadata), using the catalog’s Kerberos principal as the **proxy user**, consistent with Hive/Impala-style deployments.
Cluster admins would configure:
```xml
hadoop.proxyuser.amoro.hosts*hadoop.proxyuser.amoro.groups*hadoop.proxyuser.amoro.users*
```
(Exact host/group scope should follow cluster security policy.)
### Current implementation notes
- `TableMetaStore` already supports proxy impersonation via `doAsImpersonating(String proxyUser, Callable)`, which uses `UserGroupInformation.createProxyUser(proxyUser, catalogUgi)`.
- `AuthenticatedHadoopFileIO` wraps HDFS access with `tableMetaStore.doAs(...)` only — **not** `doAsImpersonating`.
- Trino integration is the main user-facing precedent: `mixed-format.hdfs.impersonation.enabled` + `MixedFormatHdfsAuthentication` calling `doAsImpersonating(identity.getUser(), ...)`. There is no equivalent for **AMS optimizing commit** on Hive/Iceberg catalogs.
- Optimizing commit classes (e.g. `UnKeyedTableCommit`, `KeyedTableCommit`) use table `FileIO` / `AuthenticatedFileIO.doAs()` under the catalog login user.
### Proposal
1. **Catalog-level config** (name bikeshedding welcome), e.g.
`hdfs.impersonation.enabled=true`
and optionally
`hdfs.impersonation.user-source=table-owner` (HMS owner / Iceberg table metadata).
2. When enabled, resolve the effective HDFS user per table (owner) and route:
- AMS optimizing commits (metadata avro, snapshot commit, delete files)
- Any AMS-side file cleanup tied to optimizing
- Optimizer task execution that writes replacement files (if tasks also run as service principal today)
through `TableMetaStore.doAsImpersonating(owner, ...)`.
3. **Document** required Hadoop `hadoop.proxyuser..*` settings and interaction with Ranger (authorization still applies to the **effective** user).
4. **Tests**: Kerberos or mini-cluster test with a table owned by user A and catalog principal B; commit without extra ACLs when impersonation is on.
5. **Safety**: Optional allowlist of owners/groups; clear behavior when owner is null or `hdfs`; fail fast with a actionable error if proxy is not configured.
### Why not only Ranger / ACLs?
Granting the service user global or warehouse-wide write access duplicates Hive/Impala’s **impersonation** model and increases blast radius. Owner impersonation matches how multi-tenant Hive warehouses are usually operated and avoids per-table ACL runbooks for every new table.
### References
- Hadoop proxy users: https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/Superusers.html#Configurations
- Amoro Trino doc: `mixed-format.hdfs.impersonation.enabled` (same product family, different engine)
- Code: `TableMetaStore.doAsImpersonating`, `AuthenticatedHadoopFileIO`, `MixedFormatHdfsAuthentication`
### Version
Amoro 0.9.x (behavior checked against 0.9 AMS + Iceberg Hive catalog); likely applies to earlier 0.6+ AMS deployments with Kerberos.
Contributor guide
Assessment
This issue has not been assessed yet.