pydata / pydata/xarray

Make open_zarr work with UPath

Open
#10,393 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement topic-backends
Dominant language
Python
Stars
4.2k
Forks
1.4k
Avg merge
2d 15h
Merged PRs (30d)
14

Description

Is your feature request related to a problem?

Hi xarray community.

When I try to put UPath into xr.open_zarr, it fails. I didn't check with the other open_* methods.

from upath import UPath
up = UPath('az://mycontainer@myaccount.blob.core.windows.net/mystore.zarr', anon=False)
print(up.storage_options)  # {'account_name': 'myaccount', 'anon': False}
# up.fs works fine
xr.open_zarr(up)
ValueError: unable to connect to account for Must provide either a connection_string or account_name with credentials!!

When I debugged it, the issue seems to be that xarray coerces the UPath into a string url (which is in this case just 'az://mycontainer/mystore.zarr', and then calls fsspec.url_to_fs(url, **{'asynchronous': True}), so that all storage options bound to the UPath are lost.

Describe the solution you'd like

I think it would be a very nice addition if xarray added special handling for UPath, because it seems like the perfect way to save the zarr store destination in a variable, without performing any work.

I think the implementation cannot be so naive as to just use the UPath.fs attribute as filesystem when it encounters UPath, because it needs asynchronous=True for zarr. So the solution would be to merge UPath.storage_options into the storage_options dict passed to xr.open_zarr, maybe?

I could put up a proof-of-concept PR as a first contribution if you are interested.

Describe alternatives you've considered

Everything works well if I input the url and storage options separately, but it is less convenient.

Additional context

UPath is a project by fsspec community and it seems to have quite good adoption in various python libraries.
It expands pathlib.Path interface to fsspec filesystems, such as S3. It can be used to bundle together a path, protocol and storage options in a single object, which is more convenient than passing around tuple[str, dict[str, Any]].
https://github.com/fsspec/universal_pathlib

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at xr.open_zarr and the fsspec.url_to_fs call described in the issue; trace where a UPath is coerced and where storage_options are assembled. Done means the shown Azure UPath opens successfully while preserving the asynchronous behavior needed by zarr, without requiring separate URL and storage-options arguments.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.