Add the ability to split large chunks
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 366
- Forks
- 96
- Avg merge
- 24m
- Merged PRs (30d)
- 1
Description
Currently, translating HDF5 to Zarr will result in a Zarr store with identical chunks as the source. If the source isn't chunked, this will cause worse performance when you slice a subset of the original data, since fsspec will make the full range request.
Here's a Kerchunked file
import planetary_computer
import adlfs
import kerchunk.hdf
import xarray as xr
import fsspec
credential = planetary_computer.sas.get_token("ukmeteuwest", "ukcp18").token
file = "az://ukcp18/badc/ukcp18/data/land-gcm/global/60km/rcp26/01/clt/day/v20200302/clt_rcp26_land-gcm_global_60km_01_day_18991201-19091130.nc"
storage_options = dict(account_name="ukmeteuwest", credential=credential)
file = daily_files[0]
with fsspec.open(file, **storage_options) as f:
d = kerchunk.hdf.SingleHdf5ToZarr(f, file).translate()
store = fsspec.filesystem("reference", fo=d, remote_options=storage_options).get_mapper("")
ds = xr.open_zarr(store, consolidated=False, chunks={})
ds
Timing small reads
%time ds.clt[0, 0].compute()
CPU times: user 5.54 s, sys: 1.5 s, total: 7.04 s
Wall time: 23.3 s
Compared with the non-kerchunked version
ds2 = xr.open_dataset(fsspec.open(file, **storage_options).open(), engine="h5netcdf")
%time ds2.clt[0, 0].compute()
CPU times: user 22.2 ms, sys: 8.51 ms, total: 30.7 ms
Wall time: 70.2 ms
Having the flexibility to make smaller requests by splitting large ranges into separate chunks would be helpful, if it's feasible for the backend (which it should be for these large, contiguous buffers from HDF5).
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the Kerchunked HDF5 example in the issue and measuring the small slice against the non-Kerchunked version. Then trace the HDF5-to-Zarr translation and reference-store range handling to determine where chunk splitting belongs. Done means large contiguous source ranges can be split into smaller requests without breaking existing translations.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100