Speed improvements to loading HDF5 trees
- Dominant language
- Python
- Stars
- 124
- Forks
- 46
- Avg merge
- 1h 24m
- Merged PRs (30d)
- 6
Description
In the [nexusformat API](https://github.com/nexpy/nexusformat), we load the entire HDF5 file tree by recursively walking through the groups in h5py, without reading in data values except for scalars and small arrays. On a local file, we can load files containing hundreds of objects without a significant time delay. For example, a file with 80 objects (groups, datasets, and attributes) takes 0.05s to load on my laptop. However, on h5pyd, the same load takes over 20s.
A call to load all the items in an HDF5 group requires two GET requests, and sometimes three, for each object, so there could be an improvement if all the metadata (shape, dtype, etc.) for each object were returned in a single call, and an even more significant one if all the items in a group could be returned with one GET request. Loading one group of 10 objects took 29 requests in my tests.
Binary data reads are fast, though.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.