apache / apache/arrow-rs-object-store

object_store: Automatically fall back to use_azure_cli=True if no other credentials can be found

Open
#290 13 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
322
Forks
212
Avg merge
5d 2h
Merged PRs (30d)
10

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

Currently, `object_store` automatically detects Azure credentials from environment variables and IMDS. `object_store` also supports retrieving credentials from Azure CLI, however, it does not automatically try this.
If you want to retrieve credentials from Azure CLI, you need to manually specify `use_azure_cli=True`.

**Describe the solution you'd like**

Ideally, `object_store` would try `use_azure_cli=True` if no environment or IMDS credentials can be found. This makes it possible to easily test code on your local machine (based on your Azure CLI credentials), then deploy the code unchanged to an Azure VM, and it will automatically use the environment or IMDS credentials, without requiring any code changes.

**Describe alternatives you've considered**

A workaround is to add `AZURE_USE_AZURE_CLI=True` to your local environment variables while testing. However, this is an extra step.

**Additional context**

Most Azure libraries fall back to the Azure CLI credential if environment or IMDS credentials are not found. For example, `fsspec/adlfs`, `azure.identity.DefaultAzureCredential`, `polars`, `pandas`.

I'm aware of apache/arrow-rs#6470, but I'm wondering if simply adding a fallback to an already supported authentication method is more in scope than completely replicating `DefaultAzureCredential`.

Related to this downstream `obstore` issue: https://github.com/developmentseed/obstore/issues/267#issuecomment-2660243263

Contributor guide

Open the contributing guide

Research direction

Start by reading the object_store Azure credential-selection path and tracing how environment, IMDS, and Azure CLI credentials are currently attempted. Done means Azure CLI credentials are tried only when the other supported credential sources cannot be found, while explicit configuration continues to work.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, rust
Domain
authentication, cloud
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.