Pyfive VirtualiZarr Parser?
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 104
- Forks
- 28
- Avg merge
- 22h 47m
- Merged PRs (30d)
- 6
Description
This package looks awesome. These parts of the docs particularly interested me:
The motivation for pyfive development were many, but recent developments prioritised thread-safety, lazy loading, and performance at scale in a cloud environment both standalone, and as a backend for other software such as cf-python, xarray, and h5netcdf.
We have also implemented extra methods (beyond the h5py API) to expose the chunk index directly (as well as via an iterator) and to access chunk info using the zarr indexing scheme rather than the h5py indexing scheme. This is useful for avoiding the need for a priori use of kerchunk to make a zarr index for a file.
To me it seems like the step from those features to writing a VirtualiZarr Parser is very small. We have a virtualizarr.parsers.HDFParser but it would be great to have an alternative/replacement that doesn't depend on the HDF C library. It could even live in this package if you preferred.
Is this idea of interest?
We have also implemented extra methods (beyond the h5py API) to expose the chunk index directly (as well as via an iterator) and to access chunk info using the zarr indexing scheme rather than the h5py indexing scheme. This is useful for avoiding the need for a priori use of kerchunk to make a zarr index for a file.
Also is this documented somewhere? I only found ZarrArrayStub (which seems somewhat similar to a virtualizarr.ManifestArray)
cc @sharkinsspatial @maxrjones
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing pyfive/indexing.py, especially ZarrArrayStub, and compare its exposed chunk-index behavior with virtualizarr.parsers.HDFParser and the custom parser documentation. Determine whether a pure-Python VirtualiZarr parser and documentation for the chunk index fit this package; done would require an agreed scope, implementation plan, and documented behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100