Zarr file over HTTP fetches several times the same time axis variable
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.2k
- Forks
- 1.4k
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 14
Description
What happened?
When loading a Dataset from Zarr over HTTP the time chunk is fetched 3 times from the remote location.
Example Zarr dataset (on google bucket) :
https://storage.googleapis.com/test_zarr_oie/IEA_PVPS.zarr/ADR/zarr.json
What did you expect to happen?
I expect the following URL to be fetched once only :
https://storage.googleapis.com/test_zarr_oie/IEA_PVPS.zarr/ADR/time/c/0
Instead it is fetched 3 times (logs from mitmproxy) :
16:28:07 HTTPS GET storage.googleapis.com /test_zarr_oie/IEA_PVPS.zarr/ADR/.zattrs 404 application/xml 201b 358ms
16:28:07 HTTPS GET storage.googleapis.com /test_zarr_oie/IEA_PVPS.zarr/ADR/.zgroup 404 application/xml 201b 384ms
16:28:07 HTTPS GET storage.googleapis.com /test_zarr_oie/IEA_PVPS.zarr/ADR/.zmetadata 404 application/xml 204b 346ms
16:28:07 HTTPS GET storage.googleapis.com /test_zarr_oie/IEA_PVPS.zarr/ADR/zarr.json 200 …plication/json 40.5k 204ms
16:28:07 HTTPS GET storage.googleapis.com /test_zarr_oie/IEA_PVPS.zarr/ADR/time/c/0 200 …n/octet-stream ….04k 26ms
16:28:07 HTTPS GET storage.googleapis.com /test_zarr_oie/IEA_PVPS.zarr/ADR/time/c/0 200 …n/octet-stream ….04k 24ms
16:28:07 HTTPS GET storage.googleapis.com /test_zarr_oie/IEA_PVPS.zarr/ADR/station_name/c 200 …n/octet-stream 27b 31ms
16:28:07 HTTPS GET storage.googleapis.com /test_zarr_oie/IEA_PVPS.zarr/ADR/time/c/0 200 …n/octet-stream ….04k 25ms
Minimal Complete Verifiable Example
# To spy the request, you may setup a local proxy first :
# `mitmproxy --mode reverse:https://storage.googleapis.com/ --listen-port 1111`
# Initial URL
# URL="https://storage.googleapis.com/test_zarr_oie/IEA_PVPS.zarr/ADR"
# Proxy URL
URL="http://localhost:1111/test_zarr_oie/IEA_PVPS.zarr/ADR"
ds = xr.open_dataset(URL, engine="zarr")
print(ds)
MVCE confirmation
- Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
- Complete example — the example is self-contained, including all data and the text of any traceback.
- Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
- New issue — a search of GitHub Issues suggests this is not a duplicate.
- Recent environment — the issue occurs with the latest version of xarray and its dependencies.
Relevant log output
16:28:07 HTTPS GET storage.googleapis.com /test_zarr_oie/IEA_PVPS.zarr/ADR/.zattrs 404 application/xml 201b 358ms
16:28:07 HTTPS GET storage.googleapis.com /test_zarr_oie/IEA_PVPS.zarr/ADR/.zgroup 404 application/xml 201b 384ms
16:28:07 HTTPS GET storage.googleapis.com /test_zarr_oie/IEA_PVPS.zarr/ADR/.zmetadata 404 application/xml 204b 346ms
16:28:07 HTTPS GET storage.googleapis.com /test_zarr_oie/IEA_PVPS.zarr/ADR/zarr.json 200 …plication/json 40.5k 204ms
16:28:07 HTTPS GET storage.googleapis.com /test_zarr_oie/IEA_PVPS.zarr/ADR/time/c/0 200 …n/octet-stream ….04k 26ms
16:28:07 HTTPS GET storage.googleapis.com /test_zarr_oie/IEA_PVPS.zarr/ADR/time/c/0 200 …n/octet-stream ….04k 24ms
16:28:07 HTTPS GET storage.googleapis.com /test_zarr_oie/IEA_PVPS.zarr/ADR/station_name/c 200 …n/octet-stream 27b 31ms
16:28:07 HTTPS GET storage.googleapis.com /test_zarr_oie/IEA_PVPS.zarr/ADR/time/c/0
Anything else we need to know?
No response
Environment
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the provided xr.open_dataset(URL, engine="zarr") MVCE and inspect the repeated remote request for time/c/0 while loading the example dataset. Confirm that the time chunk is fetched once rather than three times, while preserving successful dataset loading.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100