[Bug] Row-count refresh creates HMS connections without hadoop.username UGI; shared client pools then serve process-user(root) connections to queries, causing intermittent AccessControlException
- Dominant language
- Java
- Stars
- 15.9k
- Forks
- 3.9k
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 520
Description
### Search before asking
- [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues.
### Version
doris-4.0.7-rc02 (commit 35854e7e92a). Not reproducible on 4.0.5: with identical catalogs and config, fe.log shows zero ugi= HMS connections and no row-count-refresh HMS activity there.
### What's Wrong?
### Environment
- HMS-type Paimon catalog with `"hadoop.username" = "hive"` (plus an HMS `hive` catalog pointing to the same metastore)
- HiveMetastore has `hive.metastore.execute.setugi=true` + `AuthorizationPreEventListener` + `StorageBasedAuthorizationProvider`, so on `get_table` HMS performs HDFS `checkAccess` **as the client-reported user**
- Paimon warehouse dir `/paimon` is `drwx------` (accessible only to privileged users; `hive` is allowed via Ranger)
- Doris FE process runs as `root`, which has no HDFS permissions
### Symptom
Queries on Paimon tables **intermittently** fail (~4% on the incident day, in bursts of a few minutes, self-healing afterwards):
```
errCode = 2, detailMessage = failed to load paimon table .paimon.:
Failed to get Paimon table:.paimon.$null, because MetaException:
java.security.AccessControlException: Permission denied: user=root, access=EXECUTE,
inode="/paimon":paimon:hdfs:drwx------
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(...)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAccess(...)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.checkAccess(...)
```
Note `user=root` — the FE **process user** — even though the catalog sets `hadoop.username=hive`.
### Root cause analysis (source-verified on 4.0.7-rc02)
1. With `setugi=true`, connection identity is fixed once at the `set_ugi` handshake when the connection is **opened**. Wrapping later RPCs in doAs cannot change it.
2. The async row-count refresh path has **no ExecutionAuthenticator wrapping**: `ExternalRowCountCache.RowCountCacheLoader.doLoad` -> `loadRowCount` -> `ExternalTable.fetchRowCountWithMetaCache` -> `PaimonExternalTable.fetchRowCount` runs on `RowCountRefreshExecutor` threads with no doAs, so HMS clients created from this context carry the FE process user. fe.log evidence: every `RetryingMetaStoreClient ... ugi=root (auth:SIMPLE)` creation comes from `NotCheckpointRowCountRefreshExecutor-*` threads, while query threads (`mysql-nio-pool-*`) always log `ugi=hive`.
3. Paimon's `CachedClientPool` is a JVM-wide **static** cache whose key excludes UGI by default (clientClassName + metastore uris + identifier); clients are created lazily on the borrowing thread and evicted every 5 min (`client-pool-cache.eviction-interval-ms`). A connection created in a root context is later served to **properly wrapped** callers — the failing call in our incident was `PaimonExternalCatalog.getPaimonTable`, which *is* wrapped in `executionAuthenticator.execute(...)` (the `$null` suffix in the error message comes from its catch block `"$" + queryType`).
4. The same defect shape exists in `ThriftHMSCachedClient`: the inner `ThriftHMSClient` constructor calls `RetryingMetaStoreClient.getProxy(...)` **outside** the authenticator; only individual RPCs are wrapped in `ugiDoAs` — too late under `setugi`.
### Observed race timeline
Client-pool eviction at T (5-min cycle) -> ~30s later the row-count refresh thread happened to be the next connection creator -> 2 connections opened with `ugi=root` (fe.log) -> for the next minutes, queries borrowing those pooled connections failed with `Permission denied: user=root`, then the poisoned connections were evicted and everything self-healed. The bug stayed latent for 3 days after upgrading from 4.0.5, then produced 2 bursts (32 failed queries) in one day.
### Suggested fixes
1. Wrap `ExternalRowCountCache.loadRowCount` in the owning catalog's `ExecutionAuthenticator.execute(...)`.
2. Wrap `ThriftHMSClient` creation (connection open) in the authenticator — identity must bind at connection creation, not per-RPC.
3. (Paimon side) consider including `user_name` in `CachedClientPool`'s default cache key (`client-pool-cache.keys` already supports it).
### Related
- #31478 (hadoop.username not effective for filesystem Paimon catalog, fixed earlier)
- #32828 (hive catalog fails to pass hadoop username)
- Workaround we applied: inject `HADOOP_USER_NAME=hive` into the FE process environment so un-wrapped threads default to the intended identity.
### What You Expected?
Every HMS connection created by FE should carry the catalog's `hadoop.username` identity, regardless of which thread creates it. Background/statistics threads must not create connections that authenticate as the FE process user, and identity-poisoned connections must never be shared with query execution paths.
### How to Reproduce?
1. HiveMetastore with `hive.metastore.execute.setugi=true` + `hive.metastore.pre.event.listeners=AuthorizationPreEventListener` + `hive.security.metastore.authorization.manager=StorageBasedAuthorizationProvider`; Paimon warehouse dir mode 700 owned by a dedicated user (so only the configured `hadoop.username` may pass the storage check).
2. Doris FE process running as a user WITHOUT HDFS permissions (e.g. root); create an HMS-type Paimon catalog with `"hadoop.username" = ""`.
3. Query Paimon tables continuously — all succeed (query threads are correctly wrapped, connections carry the configured user).
4. Wait until a client-pool eviction (default every 5 min, `client-pool-cache.eviction-interval-ms`) coincides with the async row-count refresh thread (`RowCountRefreshExecutor`) being the next connection creator. From that moment, queries borrowing the poisoned pooled connection fail with `Permission denied: user=` for a few minutes until the next eviction. Timing dependent — in our production it stayed latent for 3 days, then produced 2 bursts / 32 failed queries in one day.
5. Grep evidence: `grep "RetryingMetaStoreClient" fe.log | grep "ugi="` — creations from `NotCheckpointRowCountRefreshExecutor-*` threads log the process user, creations from `mysql-nio-pool-*` threads log the configured `hadoop.username`. On 4.0.5 the row-count-refresh HMS activity does not exist and no process-user connections ever appear.
### Anything Else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
Contributor guide
Research direction
Trace ExternalRowCountCache.RowCountCacheLoader.doLoad through loadRowCount, ExternalTable.fetchRowCountWithMetaCache, and PaimonExternalTable.fetchRowCount, then inspect ThriftHMSCachedClient and its ThriftHMSClient constructor. Reproduce the connection-creation identity difference using the described RowCountRefreshExecutor and query-thread logs. Done means background-created HMS connections use the catalog identity and cannot poison shared client pools for queries.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100