apache / apache/amoro

Feature: HDFS commits for optimizing/maintenance as table owner (Hadoop proxy impersonation)

Open
#4,268 2 comments 0 reactions 1 assignee Claimed by @johntomcat7408-cmyk View on GitHub
Dominant language
Java
Stars
1.2k
Forks
395
Avg merge
4d 10h
Merged PRs (30d)
33

Description

### Summary

On secured Hadoop clusters, AMS and optimizers typically run as a **dedicated service principal** (e.g. `amoro`). Iceberg/Hive warehouse paths are often owned by the **table creator** (e.g. `lily`). Optimizing **tasks** can succeed while the **AMS commit** fails with HDFS `AccessControlException` on `data/` or `metadata/` because all catalog file IO runs as the service user, not the table owner.

Configuring `hadoop.proxyuser.amoro.*` on the cluster is **necessary but not sufficient**: nothing in the AMS optimizing commit path uses Hadoop proxy impersonation today.

### Environment (example)

- Amoro 0.9.x (AMS + external optimizer)
- Kerberos, Hive Metastore + Iceberg tables on HDFS
- Ranger HDFS/Hive policies for the service user
- Tables under e.g. `/warehouse/tablespace/external/hive///` with owner = end user, mode `drwxr-xr-x`

### Observed behavior

1. Optimizer finishes MINOR (or similar) work; UI shows tasks **1/1 success**.
2. AMS commit fails; optimizing process status **FAILED**.
3. AMS logs show `OptimizingCommitException` / Iceberg commit failure, root cause similar to:

`Permission denied: user=amoro, access=WRITE, inode=".../mor_events/metadata":lily:hive:drwxr-xr-x`

Ranger grants for `amoro` on `/*` and cluster `hadoop.proxyuser.amoro.{hosts,groups,users}=*` did **not** change this, because commits still execute as `amoro`.

Workarounds that work operationally but do not scale well:

- Per-table or per-DB HDFS ACLs for the service user on `data/` and `metadata/`
- Warehouse-wide default ACLs
- Treating the service user as a broad HDFS/Ranger “power user”

These are painful in multi-tenant warehouses where table ownership follows the creating user (as with Hive).

### Expected behavior

When a catalog property (or table property) enables **owner impersonation**, AMS and optimizers should perform HDFS reads/writes for optimizing commits (and related maintenance) **as the table owner** (from HMS/Iceberg metadata), using the catalog’s Kerberos principal as the **proxy user**, consistent with Hive/Impala-style deployments.

Cluster admins would configure:

```xml
hadoop.proxyuser.amoro.hosts*hadoop.proxyuser.amoro.groups*hadoop.proxyuser.amoro.users*
```

(Exact host/group scope should follow cluster security policy.)

### Current implementation notes

- `TableMetaStore` already supports proxy impersonation via `doAsImpersonating(String proxyUser, Callable)`, which uses `UserGroupInformation.createProxyUser(proxyUser, catalogUgi)`.
- `AuthenticatedHadoopFileIO` wraps HDFS access with `tableMetaStore.doAs(...)` only — **not** `doAsImpersonating`.
- Trino integration is the main user-facing precedent: `mixed-format.hdfs.impersonation.enabled` + `MixedFormatHdfsAuthentication` calling `doAsImpersonating(identity.getUser(), ...)`. There is no equivalent for **AMS optimizing commit** on Hive/Iceberg catalogs.
- Optimizing commit classes (e.g. `UnKeyedTableCommit`, `KeyedTableCommit`) use table `FileIO` / `AuthenticatedFileIO.doAs()` under the catalog login user.

### Proposal

1. **Catalog-level config** (name bikeshedding welcome), e.g.
`hdfs.impersonation.enabled=true`
and optionally
`hdfs.impersonation.user-source=table-owner` (HMS owner / Iceberg table metadata).

2. When enabled, resolve the effective HDFS user per table (owner) and route:
- AMS optimizing commits (metadata avro, snapshot commit, delete files)
- Any AMS-side file cleanup tied to optimizing
- Optimizer task execution that writes replacement files (if tasks also run as service principal today)

through `TableMetaStore.doAsImpersonating(owner, ...)`.

3. **Document** required Hadoop `hadoop.proxyuser..*` settings and interaction with Ranger (authorization still applies to the **effective** user).

4. **Tests**: Kerberos or mini-cluster test with a table owned by user A and catalog principal B; commit without extra ACLs when impersonation is on.

5. **Safety**: Optional allowlist of owners/groups; clear behavior when owner is null or `hdfs`; fail fast with a actionable error if proxy is not configured.

### Why not only Ranger / ACLs?

Granting the service user global or warehouse-wide write access duplicates Hive/Impala’s **impersonation** model and increases blast radius. Owner impersonation matches how multi-tenant Hive warehouses are usually operated and avoids per-table ACL runbooks for every new table.

### References

- Hadoop proxy users: https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/Superusers.html#Configurations
- Amoro Trino doc: `mixed-format.hdfs.impersonation.enabled` (same product family, different engine)
- Code: `TableMetaStore.doAsImpersonating`, `AuthenticatedHadoopFileIO`, `MixedFormatHdfsAuthentication`

### Version

Amoro 0.9.x (behavior checked against 0.9 AMS + Iceberg Hive catalog); likely applies to earlier 0.6+ AMS deployments with Kerberos.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.