fsspec / fsspec/kerchunk

Combining ReferenceFileSystem, dask, and async requests for HDF data

Open
#14 17 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
366
Forks
96
Avg merge
24m
Merged PRs (30d)
1

Description

https://medium.com/pangeo/cloud-performant-netcdf4-hdf5-with-zarr-fsspec-and-intake-3d3a3e7cb935
This blog post is great! There is huge potential in using unmodified HDF files in object storage and obtaining better read efficiency with fsspec and sidecar metadata.

Often existing HDF data has small chunks (the example in the blog post suggests an HDF chunk size of 10MB, but then uses dask with dask chunks specified as 30MB to speed up computations). It's not uncommon to come across files written with 1MB chunks either! But here are screenshots of the two cases from the blog post:

1. Dask chunks = HDF chunks:
image

2. Dask chunks > HDF chunks
image

It's unclear to me if fsspec async requests come into play in the case where dask chunks>hdf chunks by default as suggested in this forum post (https://discourse.pangeo.io/t/understanding-async/1098/6). It seems like there would be a major performance boost in that case.

Perhaps @rsignell-usgs , @martindurant , @ajelenak , or @rabernat can clarify?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the linked blog post and the Pangeo forum discussion about asynchronous requests. Determine whether async access improves reads when Dask chunks exceed HDF chunks, and identify the project entry points or tests that would need to demonstrate that behavior. Done should include a decided scope and documented or validated performance guidance.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.