[FEATURE REQUEST] Add option to use environment variables for ADLS Subscoped credentials
- Dominant language
- Java
- Stars
- 2.1k
- Forks
- 522
- Avg merge
- 1d 17h
- Merged PRs (30d)
- 137
Description
**Is your feature request related to a problem? Please describe.**
My organization does not allow getting user delegation key in Azure and the only option for us to authenticate with Azure is to use service principal. When my spark job tries to write to Azure, I will get the following exception at Polaris server:
```
c.a.s.f.d.DataLakeServiceClient: If you are using a StorageSharedKeyCredential, and the server returned an error message that says 'Signature did not match', you can compare the string to sign with the one generated by the SDK. To log the string to sign, pass in the context key value pair 'Azure-Storage-Log-String-To-Sign': true to the appropriate method call. If you are using a SAS token, and the server returned an error message that says 'Signature did not match', you can compare the string to sign with the one generated by the SDK. To log the string to sign, pass in the context key value pair 'Azure-Storage-Log-String-To-Sign': true to the appropriate generateSas method call. Please remember to disable 'Azure-Storage-Log-String-To-Sign' before going to production as this string can potentially contain PII."
```
```xml
AuthorizationPermissionMismatchThis request is not authorized to perform this operation using this permission."
```
**Describe the solution you'd like**
Since `ADLSFileIO` fall back to use `DefaultAzureCredentialBuilder` when there are no `SAS token` or `Storage Shared Key credential`, we can have a default catalog option to use environment as authentication type e.g.:
```json
{
"name": "test",
"type": "INTERNAL",
"properties": {
"default-base-location": "abfss://container@storageaccount.dfs.core.windows.net/test/"
},
"storageConfigInfo": {
"tenantId": "tenant-id",
"storageType": "AZURE",
"allowedLocations": [
"abfss://container@storageaccount.dfs.core.windows.net/test/"
],
"authType": "APPLICATION_DEFAULT"
}
}
```
so that any query engine can abstract away the credential and the main credential still being govern by Polaris Catalog for Azure.
Contributor guide
Research direction
Start by tracing ADLSFileIO and its DefaultAzureCredentialBuilder fallback, then inspect how storageConfigInfo and authType are handled for Azure catalogs. Done means an APPLICATION_DEFAULT option can use environment-based credentials while credential use remains governed by the Polaris catalog configuration.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, java
- Domain
- authentication, cloud
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100