apache / apache/airflow

Azure Datalake Storage V2 ObjectStoragePath connection issues

Open
#40,410 5 comments 0 reactions 0 assignees View on GitHub
area:providers good first issue kind:bug provider:microsoft-azure
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

### Apache Airflow version

2.9.2

### If "Other Airflow 2 version" selected, which one?

_No response_

### What happened?

Connection parsing seems buggy with the Azure implementation for ObjectStoragePath - requiring specific extras in specific places that don't really make sense. This is also inconsistent with the `AzureDataLakeStorageV2Hook` connection parsing

Additionally - there is no documentation at all about an Azure implementation for ObjectStoragePath - so we should make sure to have a doc associated with the provider.

Furthermore, this is a Microsoft problem - but why there are [three solutions for the same thing, each with different terminology](https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-best-practices#understand-the-terms-used-in-documentation), in varying degrees of supported or deprecated - is wicked confusing.

### What you think should happen instead?

_No response_

### How to reproduce

1) ✅ `extras.connection_string` - works for both the Hook and Object Storage, without issue:
```
import os
os.environ["AIRFLOW_CONN_ADLS"] = '{"conn_type": "adls", "extra": {"connection_string": "..."}}'

from airflow.providers.microsoft.azure.fs.adls import get_fs
from airflow.providers.microsoft.azure.hooks.data_lake import AzureDataLakeStorageV2Hook

# Hook
print(list(AzureDataLakeStorageV2Hook(adls_conn_id="ADLS").get_conn().list_file_systems()))
# Object Storage
get_fs("ADLS")
```

2) ❌ `host`+`login`+`password`+`extras.tenant_id` - ✅ works for the Hook, ❌ DOES NOT WORK for Object Storage:
```
import os
os.environ["AIRFLOW_CONN_ADLS"] = '{"conn_type": "adls", "host": "myfilesystem", "login": "...", "password": "...", "extra": {"tenant_id": "..."}}'

from airflow.providers.microsoft.azure.fs.adls import get_fs
from airflow.providers.microsoft.azure.hooks.data_lake import AzureDataLakeStorageV2Hook

# Hook
print(list(AzureDataLakeStorageV2Hook(adls_conn_id="ADLS").get_conn().list_file_systems()))
# Object Storage
get_fs("ADLS")
# ValueError: unable to connect to account for Must provide either a connection_string or account_name with credentials!!
```
Error from `adlfs.spec@do_connect`

3) ❌ `host`+`login`+`password`+`extras.tenant_id`+`extras.account_name` (not [documented](https://airflow.apache.org/docs/apache-airflow-providers-microsoft-azure/stable/connections/adls_v2.html)). ~~Works for both~~
```
import os
os.environ["AIRFLOW_CONN_ADLS"] = '{"conn_type": "adls", "host": "myfilesystem", "login": "...", "password": "...", "extra": {"tenant_id": "...", "account_name": "myfilesystem"}}'

from airflow.providers.microsoft.azure.fs.adls import get_fs
from airflow.providers.microsoft.azure.hooks.data_lake import AzureDataLakeStorageV2Hook

# Hook
print(list(AzureDataLakeStorageV2Hook(adls_conn_id="ADLS").get_conn().list_file_systems()))
# Object Storage
get_fs("ADLS")

# test #2
get_fs("ADLS").ls("/")
# ClientAuthenticationError: Server failed to authenticate the request. Please refer to the information in the ww-authentication header.
# 'WWW-Authenticate': 'Bearer authorization_uri=https://login.microsoftonline.com/.../oauth2/authorize resource_id=https://storage.azure.com"
```
(edit: I initially thought this was working, as `get_fs` returns successfully, but as soon as I attempt to use it it fails. I've tried a number of other combinations, such as including `account_url` and `client_secret_auth_config` in `extra` - none are working)

### Operating System

Astronomer/Docker

### Versions of Apache Airflow Providers

_No response_

### Deployment

Astronomer

### Deployment details

_No response_

### Anything else?

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start with airflow.providers.microsoft.azure.fs.adls.get_fs and AzureDataLakeStorageV2Hook in hooks/data_lake, comparing how each parses the connection examples in the issue. Review the existing adls_v2 connection documentation, then reproduce the credential combinations and determine consistent ObjectStoragePath behavior. Done means the supported Azure connection forms work for both interfaces and the provider documents them clearly.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, python
Domain
cloud, data-engineering, documentation
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.