Azure Datalake Storage V2 ObjectStoragePath connection issues
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 483
Description
### Apache Airflow version
2.9.2
### If "Other Airflow 2 version" selected, which one?
_No response_
### What happened?
Connection parsing seems buggy with the Azure implementation for ObjectStoragePath - requiring specific extras in specific places that don't really make sense. This is also inconsistent with the `AzureDataLakeStorageV2Hook` connection parsing
Additionally - there is no documentation at all about an Azure implementation for ObjectStoragePath - so we should make sure to have a doc associated with the provider.
Furthermore, this is a Microsoft problem - but why there are [three solutions for the same thing, each with different terminology](https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-best-practices#understand-the-terms-used-in-documentation), in varying degrees of supported or deprecated - is wicked confusing.
### What you think should happen instead?
_No response_
### How to reproduce
1) ✅ `extras.connection_string` - works for both the Hook and Object Storage, without issue:
```
import os
os.environ["AIRFLOW_CONN_ADLS"] = '{"conn_type": "adls", "extra": {"connection_string": "..."}}'
from airflow.providers.microsoft.azure.fs.adls import get_fs
from airflow.providers.microsoft.azure.hooks.data_lake import AzureDataLakeStorageV2Hook
# Hook
print(list(AzureDataLakeStorageV2Hook(adls_conn_id="ADLS").get_conn().list_file_systems()))
# Object Storage
get_fs("ADLS")
```
2) ❌ `host`+`login`+`password`+`extras.tenant_id` - ✅ works for the Hook, ❌ DOES NOT WORK for Object Storage:
```
import os
os.environ["AIRFLOW_CONN_ADLS"] = '{"conn_type": "adls", "host": "myfilesystem", "login": "...", "password": "...", "extra": {"tenant_id": "..."}}'
from airflow.providers.microsoft.azure.fs.adls import get_fs
from airflow.providers.microsoft.azure.hooks.data_lake import AzureDataLakeStorageV2Hook
# Hook
print(list(AzureDataLakeStorageV2Hook(adls_conn_id="ADLS").get_conn().list_file_systems()))
# Object Storage
get_fs("ADLS")
# ValueError: unable to connect to account for Must provide either a connection_string or account_name with credentials!!
```
Error from `adlfs.spec@do_connect`
3) ❌ `host`+`login`+`password`+`extras.tenant_id`+`extras.account_name` (not [documented](https://airflow.apache.org/docs/apache-airflow-providers-microsoft-azure/stable/connections/adls_v2.html)). ~~Works for both~~
```
import os
os.environ["AIRFLOW_CONN_ADLS"] = '{"conn_type": "adls", "host": "myfilesystem", "login": "...", "password": "...", "extra": {"tenant_id": "...", "account_name": "myfilesystem"}}'
from airflow.providers.microsoft.azure.fs.adls import get_fs
from airflow.providers.microsoft.azure.hooks.data_lake import AzureDataLakeStorageV2Hook
# Hook
print(list(AzureDataLakeStorageV2Hook(adls_conn_id="ADLS").get_conn().list_file_systems()))
# Object Storage
get_fs("ADLS")
# test #2
get_fs("ADLS").ls("/")
# ClientAuthenticationError: Server failed to authenticate the request. Please refer to the information in the ww-authentication header.
# 'WWW-Authenticate': 'Bearer authorization_uri=https://login.microsoftonline.com/.../oauth2/authorize resource_id=https://storage.azure.com"
```
(edit: I initially thought this was working, as `get_fs` returns successfully, but as soon as I attempt to use it it fails. I've tried a number of other combinations, such as including `account_url` and `client_secret_auth_config` in `extra` - none are working)
### Operating System
Astronomer/Docker
### Versions of Apache Airflow Providers
_No response_
### Deployment
Astronomer
### Deployment details
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
Contributor guide
Research direction
Start with airflow.providers.microsoft.azure.fs.adls.get_fs and AzureDataLakeStorageV2Hook in hooks/data_lake, comparing how each parses the connection examples in the issue. Review the existing adls_v2 connection documentation, then reproduce the credential combinations and determine consistent ObjectStoragePath behavior. Done means the supported Azure connection forms work for both interfaces and the provider documents them clearly.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, python
- Domain
- cloud, data-engineering, documentation
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100