apache / apache/datafusion-comet

Native Azure store lets ambient AZURE_* environment variables override or corrupt explicit Hadoop auth config

Open
#5,542 1 comment 0 reactions 0 assignees View on GitHub
area:scan bug priority:high
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 6h
Merged PRs (30d)
190

Description

### Describe the bug

Native Azure store construction (azure.rs create_store) seeds the builder with MicrosoftAzureBuilder::from_env() and then layers the translated fs.azure.* config on top. Same-key collisions resolve correctly (Hadoop wins), but object_store's build() selects the credential by a fixed chain across different keys — Bearer > AccessKey > WorkloadIdentity > ClientSecret > SAS > CLI > IMDS — so ambient env vars that land higher in the chain than the configured mechanism win. Hadoop's ABFS driver reads no environment variables at all (verified: zero getenv references across all classes of hadoop-azure 3.3.4 and 3.4.1), so the two engines can resolve different identities for the same table.

This is the flip side of the #4747 fix, which added the env pickup to make workload identity work.

### Steps to reproduce

No Azure account needed; the credential the builder selects was extracted directly from the store built by the exact create_store path, with controlled env + Hadoop conf:

1. env AZURE_STORAGE_TOKEN=ambient-env-bearer-token + Hadoop fs.azure.account.key. set → winning credential: BearerToken("ambient-env-bearer-token"). The configured account key is ignored.
2. env AZURE_STORAGE_ACCOUNT_KEY=env-injected-key + Hadoop OAuth client-credentials configured → winning credential: AccessKey("env-injected-key"). The configured service principal is ignored.
3. env AZURE_FEDERATED_TOKEN_FILE= (what the AKS Workload Identity webhook injects into every annotated pod) + Hadoop OAuth client id/tenant/secret configured → winning credential: a hybrid WorkloadIdentityOAuth using Hadoop's client-id and tenant with the webhook's token file; the explicitly configured client secret is silently dropped.

Controls: same-key collisions correctly resolve to the Hadoop value, and env-only workload identity works as #4747 intended. Also latent: AZURE_STORAGE_USE_EMULATOR, AZURE_ALLOW_HTTP, and AZURE_PROXY_URL parse from env and reroute transport.

On a real cluster this surfaces as a 403 on tables the stock reader handles fine or, when the ambient identity also has access, a read under a credential the job never configured.

### Expected behavior

Explicit fs.azure.* auth configuration should fully determine the native store's identity, exactly as it does for Hadoop's own reader. Ambient env should only be consulted when Hadoop config specifies no auth mechanism (preserving the #4747 workload-identity case).

### Additional context

Fix directions: when the translated Hadoop config carries an auth mechanism, clear/skip conflicting env-derived keys before build(); or gate admission like the S3/GCS credential-divergence checks in #5365. The Delta contrib is currently unaffected (it declines all abfss:// scans, see #4993); the exposed path is plain parquet on abfss.

Contributor guide

Open the contributing guide

Research direction

Start in azure.rs at create_store and trace how MicrosoftAzureBuilder::from_env() is combined with translated fs.azure.* configuration. Reproduce the controlled environment/Hadoop configuration cases, then verify that explicit Hadoop authentication determines the native store identity while env-only workload identity still works as intended by #4747.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, hadoop, rust
Domain
authentication, backend, cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
62/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.