fsspec / fsspec/kerchunk

Add the ability to split large chunks

Open
#124 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
366
Forks
96
Avg merge
24m
Merged PRs (30d)
1

Description

Currently, translating HDF5 to Zarr will result in a Zarr store with identical chunks as the source. If the source isn't chunked, this will cause worse performance when you slice a subset of the original data, since fsspec will make the full range request.

Here's a Kerchunked file

import planetary_computer
import adlfs
import kerchunk.hdf
import xarray as xr
import fsspec

credential = planetary_computer.sas.get_token("ukmeteuwest", "ukcp18").token
file = "az://ukcp18/badc/ukcp18/data/land-gcm/global/60km/rcp26/01/clt/day/v20200302/clt_rcp26_land-gcm_global_60km_01_day_18991201-19091130.nc"

storage_options = dict(account_name="ukmeteuwest", credential=credential)
file = daily_files[0]
with fsspec.open(file, **storage_options) as f:
    d = kerchunk.hdf.SingleHdf5ToZarr(f, file).translate()

store = fsspec.filesystem("reference", fo=d, remote_options=storage_options).get_mapper("")

ds = xr.open_zarr(store, consolidated=False, chunks={})
ds

Timing small reads

%time ds.clt[0, 0].compute()
CPU times: user 5.54 s, sys: 1.5 s, total: 7.04 s
Wall time: 23.3 s

Compared with the non-kerchunked version

ds2 = xr.open_dataset(fsspec.open(file, **storage_options).open(), engine="h5netcdf")

%time ds2.clt[0, 0].compute()
CPU times: user 22.2 ms, sys: 8.51 ms, total: 30.7 ms
Wall time: 70.2 ms

Having the flexibility to make smaller requests by splitting large ranges into separate chunks would be helpful, if it's feasible for the backend (which it should be for these large, contiguous buffers from HDF5).

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the Kerchunked HDF5 example in the issue and measuring the small slice against the non-Kerchunked version. Then trace the HDF5-to-Zarr translation and reference-store range handling to determine where chunk splitting belongs. Done means large contiguous source ranges can be split into smaller requests without breaking existing translations.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.