Combining ReferenceFileSystem, dask, and async requests for HDF data
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 366
- Forks
- 96
- Avg merge
- 24m
- Merged PRs (30d)
- 1
Description
https://medium.com/pangeo/cloud-performant-netcdf4-hdf5-with-zarr-fsspec-and-intake-3d3a3e7cb935
This blog post is great! There is huge potential in using unmodified HDF files in object storage and obtaining better read efficiency with fsspec and sidecar metadata.
Often existing HDF data has small chunks (the example in the blog post suggests an HDF chunk size of 10MB, but then uses dask with dask chunks specified as 30MB to speed up computations). It's not uncommon to come across files written with 1MB chunks either! But here are screenshots of the two cases from the blog post:
1. Dask chunks = HDF chunks:

2. Dask chunks > HDF chunks

It's unclear to me if fsspec async requests come into play in the case where dask chunks>hdf chunks by default as suggested in this forum post (https://discourse.pangeo.io/t/understanding-async/1098/6). It seems like there would be a major performance boost in that case.
Perhaps @rsignell-usgs , @martindurant , @ajelenak , or @rabernat can clarify?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the linked blog post and the Pangeo forum discussion about asynchronous requests. Determine whether async access improves reads when Dask chunks exceed HDF chunks, and identify the project entry points or tests that would need to demonstrate that behavior. Done should include a decided scope and documented or validated performance guidance.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100