Nested HDF5 Data / HEC-RAS
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 366
- Forks
- 96
- Avg merge
- 24m
- Merged PRs (30d)
- 1
Description
I'm working on development of the rashdf library for reading HEC-RAS HDF5 data. A big part of the motivation for development of the library is stochastic hydrologic/hydraulic modeling.
We want to be able to generate Zarr metadata for stochastic HEC-RAS outputs, so that e.g., results for many different stochastic flood simulations from a given RAS model can be opened as a single xarray Dataset. For example, results for 100 different simulations could be concatenated in a new simulation dimension, with coordinates being the index number of each simulation. It took me a little while to figure out how to make that happen because RAS HDF5 data is highly nested and doesn't conform to typical conventions.
The way I implemented it is hacky:
- Given an
xr.Datasetpulled from the HDF file and the path of each childxr.DataArraywithin the HDF file, - Get the filters for each DataArray:
filters = SingleHdf5ToZarr._decode_filters(None, hdf_ds) - Get the storage info for each DataArray:
storage_info = SingleHdf5ToZarr._storage_info(None, hdf_ds) - Build out metadata for chunks using
storage_info - "Write" the
xr.Datasetto azarr.MemoryStorewithcompute=False, to generate the framework of what's needed for the Zarr metadata - Read the objects generated by writing to
zarr.MemoryStoreand decode - Assemble the
zarr.MemoryStoreobjects,filters, andstorage_infointo a dictionary and finally return
I suppose my questions are:
- Is there a better way to approach highly nested or otherwise idiosyncratic HDF5 data with Kerchunk?
- Could Kerchunk's
SingleHdf5ToZarr._decode_filtersand_storage_infomethods be made public?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the linked rashdf pull request #59 and the SingleHdf5ToZarr._decode_filters and _storage_info entry points mentioned in the issue. Determine whether nested HDF5 data needs a supported public path, and define what API behavior and documentation would demonstrate completion.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100