Cannot read Zarr v2 data from SWIFT without consolidated metadata
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.2k
- Forks
- 1.4k
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 14
Description
What happened?
I'm trying to work with Zarr v2 data on a SWIFT store, with a fresh install of xarray and zarr in a virtual environment.
pip install swiftspec xarray "zarr[remote]"
If I open a dataset (which has been tested and used previously) without using the consolidated metadata, no variables are found:
import xarray as xr
botany_url="https://swift.dkrz.de/v1/dkrz_b42c6bc6-439f-4efb-9491-db3de70ad9ba/botany_1536_2D/botany_1536_2D.zarr"
ds_2D = xr.open_zarr(botany_url,consolidated=False)
print(ds_2D)
Changing to consolidated=True shows the variables. The dataset can be browsed here.
A possibly related issue:
When writing a Zarr v2 dataset to a SWIFT store, the consolidated metadata is written correctly if the dimension separator is the default, ., but not if it is changed to /. This means a v2 dataset with separator / written to SWIFT cannot be read back. This may be specific to SWIFT or more broadly to fsspec stores - writing to a local directory with a zarr.storage.DirectoryStore does write the consolidated metadata correctly (but this was tested with zarr 2.18.7 because in zarr 3 the DirectoryStore is gone).
Third issue: how do I write a Zarr v2 dataset with dimension_separator='/' using the current versions of everything, including zarr v3.x?
What did you expect to happen?
I expect to be able to read Zarr v2 from a SWIFT store with or without consolidated metadata.
I expect to be able to write Zarr v2 with dimension_separator='/' using the current versions of all the modules.
Minimal Complete Verifiable Example
import xarray as xr
botany_url="https://swift.dkrz.de/v1/dkrz_b42c6bc6-439f-4efb-9491-db3de70ad9ba/botany_1536_2D/botany_1536_2D.zarr"
ds_2D = xr.open_zarr(botany_url,consolidated=False)
print(ds_2D)
MVCE confirmation
- Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
- Complete example — the example is self-contained, including all data and the text of any traceback.
- Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
- New issue — a search of GitHub Issues suggests this is not a duplicate.
- Recent environment — the issue occurs with the latest version of xarray and its dependencies.
Relevant log output
Anything else we need to know?
No response
Environment
INSTALLED VERSIONS
commit: None
python: 3.12.3 (main, Jun 18 2025, 17:59:45) [GCC 13.3.0]
python-bits: 64
OS: Linux
OS-release: 6.8.0-63-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: ('en_US', 'UTF-8')
libhdf5: None
libnetcdf: None
xarray: 2025.7.1
pandas: 2.3.1
numpy: 2.3.2
scipy: None
netCDF4: None
pydap: None
h5netcdf: None
h5py: None
zarr: 3.1.0
cftime: None
nc_time_axis: None
iris: None
bottleneck: None
dask: None
distributed: None
matplotlib: None
cartopy: None
seaborn: None
numbagg: None
fsspec: 2025.7.0
cupy: None
pint: None
sparse: None
flox: None
numpy_groupies: None
setuptools: None
pip: 24.0
conda: None
pytest: None
mypy: None
IPython: None
sphinx: None
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the provided xr.open_zarr minimal example and trace how consolidated metadata, SWIFT access, and Zarr v2 dimension_separator='/' are handled. Reproduce the read and write cases against the linked SWIFT dataset; done means establishing whether both non-consolidated reads and slash-separated Zarr v2 writes work with the current dependency versions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100