lance-format / lance-format/lance

docs: document supported Azure file path formats

Open
#6,663 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

A-docs documentation
Dominant language
Rust
Stars
7.1k
Forks
852
Avg merge
3d 18h
Merged PRs (30d)
272

Description

The object_store crate (apache/arrow-rs-object-store) supports a variety of Azure URL/path formats in its Azure builder. We should clearly document which formats Lance officially supports, so users know what they can pass as dataset URIs.

Here are all the variations recognized by the upstream object_store Azure builder:

Scheme-based (fsspec / Hadoop style):

  • abfs[s]://<container>/<path>
  • abfs[s]://<file_system>@<account_name>.dfs.core.windows.net/<path>
  • abfs[s]://<file_system>@<account_name>.dfs.fabric.microsoft.com/<path>
  • az://<container>/<path>
  • adl://<container>/<path>

HTTPS (Azure Blob Storage):

  • https://<account>.blob.core.windows.net
  • https://<account>.blob.core.windows.net/<container>
  • https://<account>.dfs.core.windows.net

HTTPS (Microsoft Fabric / OneLake):

  • https://<account>.dfs.fabric.microsoft.com
  • https://<account>.dfs.fabric.microsoft.com/<container>
  • https://<account>.blob.fabric.microsoft.com
  • https://<account>.blob.fabric.microsoft.com/<container>
  • https://onelake.dfs.fabric.microsoft.com/<workspaceGUID>/<itemGUID>/Files/<path>
  • https://onelake.dfs.fabric.microsoft.com/<workspace>/<item>.<itemtype>/<path>/<fileName>

We should:

  1. Document which of these formats are officially supported and tested in Lance (Rust, Python, Java).
  2. Call out any formats that are known to not work or have caveats.
  3. Add examples in our documentation showing common Azure usage patterns.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the object_store Azure builder and checking the listed URI variations across Lance's Rust, Python, and Java interfaces. Record which formats are supported and tested, note known failures or caveats, and add documentation examples for common Azure usage patterns; done means the supported formats and limitations are clear to users.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, java, python, rust
Domain
cloud, documentation
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.