NCAS-CMS / NCAS-CMS/pyfive

Pyfive VirtualiZarr Parser?

Open
#155 1 comment 3 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Python
Stars
104
Forks
28
Avg merge
22h 47m
Merged PRs (30d)
6

Description

This package looks awesome. These parts of the docs particularly interested me:

The motivation for pyfive development were many, but recent developments prioritised thread-safety, lazy loading, and performance at scale in a cloud environment both standalone, and as a backend for other software such as cf-python, xarray, and h5netcdf.

We have also implemented extra methods (beyond the h5py API) to expose the chunk index directly (as well as via an iterator) and to access chunk info using the zarr indexing scheme rather than the h5py indexing scheme. This is useful for avoiding the need for a priori use of kerchunk to make a zarr index for a file.

To me it seems like the step from those features to writing a VirtualiZarr Parser is very small. We have a virtualizarr.parsers.HDFParser but it would be great to have an alternative/replacement that doesn't depend on the HDF C library. It could even live in this package if you preferred.

Is this idea of interest?

We have also implemented extra methods (beyond the h5py API) to expose the chunk index directly (as well as via an iterator) and to access chunk info using the zarr indexing scheme rather than the h5py indexing scheme. This is useful for avoiding the need for a priori use of kerchunk to make a zarr index for a file.

Also is this documented somewhere? I only found ZarrArrayStub (which seems somewhat similar to a virtualizarr.ManifestArray)

cc @sharkinsspatial @maxrjones

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing pyfive/indexing.py, especially ZarrArrayStub, and compare its exposed chunk-index behavior with virtualizarr.parsers.HDFParser and the custom parser documentation. Determine whether a pure-Python VirtualiZarr parser and documentation for the chunk index fit this package; done would require an agreed scope, implementation plan, and documented behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.