apache / apache/polaris

[FEATURE REQUEST] Add option to use environment variables for ADLS Subscoped credentials

Open
#69 2 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
2.1k
Forks
522
Avg merge
1d 17h
Merged PRs (30d)
137

Description

**Is your feature request related to a problem? Please describe.**
My organization does not allow getting user delegation key in Azure and the only option for us to authenticate with Azure is to use service principal. When my spark job tries to write to Azure, I will get the following exception at Polaris server:
```
c.a.s.f.d.DataLakeServiceClient: If you are using a StorageSharedKeyCredential, and the server returned an error message that says 'Signature did not match', you can compare the string to sign with the one generated by the SDK. To log the string to sign, pass in the context key value pair 'Azure-Storage-Log-String-To-Sign': true to the appropriate method call. If you are using a SAS token, and the server returned an error message that says 'Signature did not match', you can compare the string to sign with the one generated by the SDK. To log the string to sign, pass in the context key value pair 'Azure-Storage-Log-String-To-Sign': true to the appropriate generateSas method call. Please remember to disable 'Azure-Storage-Log-String-To-Sign' before going to production as this string can potentially contain PII."
```
```xml
AuthorizationPermissionMismatchThis request is not authorized to perform this operation using this permission."
```

**Describe the solution you'd like**
Since `ADLSFileIO` fall back to use `DefaultAzureCredentialBuilder` when there are no `SAS token` or `Storage Shared Key credential`, we can have a default catalog option to use environment as authentication type e.g.:
```json
{
"name": "test",
"type": "INTERNAL",
"properties": {
"default-base-location": "abfss://container@storageaccount.dfs.core.windows.net/test/"
},
"storageConfigInfo": {
"tenantId": "tenant-id",
"storageType": "AZURE",
"allowedLocations": [
"abfss://container@storageaccount.dfs.core.windows.net/test/"
],
"authType": "APPLICATION_DEFAULT"
}
}
```
so that any query engine can abstract away the credential and the main credential still being govern by Polaris Catalog for Azure.

Contributor guide

Open the contributing guide

Research direction

Start by tracing ADLSFileIO and its DefaultAzureCredentialBuilder fallback, then inspect how storageConfigInfo and authType are handled for Azure catalogs. Done means an APPLICATION_DEFAULT option can use environment-based credentials while credential use remains governed by the Polaris catalog configuration.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, java
Domain
authentication, cloud
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.