`remove_dir_all` does not delete directories on ADLS Gen2 with hierarchical namespace

What happened?

ObjectStore::remove_dir_all in lance-io fails to fully remove a dataset path on Azure Data Lake Storage Gen2 when hierarchical namespace (HNS) is enabled and the default MicrosoftAzure backend is in use. All blobs are deleted, but the empty directory tree remains.

What was the expected behavior?

After remove_dir_all completes, the target path should not exist — including the directory itself. Operations like Dataset.drop() should leave no trace on storage.

How to reproduce?

1. Create a Lance dataset on ADLS Gen2 with hierarchical namespace enabled (abfss:// scheme)
2. Call Dataset.drop(uri, storageOptions) (Java) or equivalent Rust path (without use_opendal=true)
3. Observe: all blobs under the path are deleted, but the empty directory tree still exists

Root Cause

remove_dir_all uses list() + delete_stream() to enumerate and delete objects. On the default (non-OpenDAL) Azure path, the object_store crate's MicrosoftAzure backend unconditionally filters directory entries from list() results:

// object_store 0.13.2, azure/client.rs
.filter(|blob| {
    !matches!(blob.properties.resource_type.as_ref(), Some(typ) if typ == "directory")
})

Since directories are never returned by list(), they are never passed to delete_stream(). After all blobs are removed, the empty directory tree persists as real HNS filesystem entities.

On flat object stores (S3, GCS, Azure Blob without HNS), this works because directories are virtual — shared key prefixes that vanish when all objects under them are deleted. On ADLS Gen2 with HNS, directories exist independently and must be explicitly deleted.

Why the Default Backend Uses the Blob Endpoint

MicrosoftAzureBuilder::build() constructs the service URL as https://{account}.blob.core.windows.net by default — even when the input URI scheme is abfss://. The builder's parse_url for abfss:// extracts only account/container information; it does not switch the API endpoint. The Blob endpoint has no concept of directories as addressable entities — only the DFS endpoint (dfs.core.windows.net) natively understands HNS directories.

When This Does NOT Occur

When use_opendal=true is passed in storage options, lance uses OpenDAL's Azdls backend which targets the DFS endpoint directly. Its list returns directories as entries, and its delete issues requests to the DFS API that can remove empty directories.

Why use_opendal=true Cannot Be the Default Today

The MicrosoftAzure path integrates with StorageOptionsAccessor for dynamic credential refresh — credentials (e.g., SAS tokens) are rotated at runtime without rebuilding the store. The OpenDAL path currently uses static/environment-backed credentials only (this is an integration choice, not a framework limitation).

However, the infrastructure to solve this already exists in the codebase: DynamicOpenDalStore (used by HuggingFace, OSS, and TOS providers) wraps any OpendalStore and provides automatic credential refresh by calling StorageOptionsAccessor::get_storage_options() before each operation, rebuilding the underlying operator when the normalized config changes. Wiring this into the Azure OpenDAL path is integration work, not new design.

Proposed Roadmap

Phase 1: Wire dynamic credentials into the OpenDAL Azure path using the existing DynamicOpenDalStore pattern (already proven for OSS/TOS/HuggingFace). This requires:

• Passing the StorageOptionsAccessor to build_opendal_azure_store()
• Constructing a DynamicOpenDalStore with normalize_opendal_azure_options as the normalization function
• Protecting structural keys (filesystem, endpoint, root) from dynamic override

This is integration work (~30-50 lines), not new design.

Phase 2: Make Azdls the default backend for abfss://. This fixes directory deletion, enables atomic DFS rename, and provides correct ACL propagation. A use_legacy_blob_endpoint=true escape hatch can be provided.

Migration risks to address:

• SAS token scoping: Tokens vended for blob.core.windows.net are invalid against dfs.core.windows.net. Credential providers must be updated to vend DFS-scoped tokens before or alongside this switch.
• Error codes and retry behavior: DFS REST API returns different HTTP status codes for the same logical errors; retry/throttle logic may need adjustment.
• Performance characteristics: Blob and DFS endpoints have separate scaling/throttling limits per storage account.

Phase 3: After production soak with telemetry confirming adoption and no regressions, deprecate and remove the MicrosoftAzureBuilder path for abfss:// URIs (it remains the default for az://). Prerequisites:

• Confirmed that all known credential providers can vend DFS-scoped tokens
• Sufficient adoption of the new path in production
• No reported regressions in error handling or performance

Interim Workaround

• Pass use_opendal=true in storage options (if dynamic credential refresh is not needed)
• Perform directory cleanup externally after remove_dir_all (e.g., via a DFS REST call or Hadoop ABFS FileSystem)

Environment

• Lance version: 9.1.0-beta.2
• object_store crate version: 0.13.2
• opendal version: 0.57.0
• Storage: Azure Data Lake Storage Gen2 with hierarchical namespace enabled
• URI scheme: abfss://

Start by reading ObjectStore::remove_dir_all, build_opendal_azure_store, DynamicOpenDalStore, and normalize_opendal_azure_options, then trace the default Azure path for abfss://. Clarify which roadmap phase is in scope and identify existing tests for Azure deletion and credential refresh. Done means removing an HNS dataset path without leaving directories while preserving dynamic credential refresh.