HDFGroup / HDFGroup/hdf5

Improve float16 performance

Open
#2,154 4 comments 0 reactions 1 assignee Assigned to @jhendersonHDF View on GitHub
Component - C Library
Dominant language
C
Stars
988
Forks
355
Avg merge
4d 2h
Merged PRs (30d)
12

Description

Using HDF5 to read data stored as 16-bit floating point into a 32-bit buffer is extremely slow, around 16x slower than an equivalent conversion in numpy. I uploaded a demo [here](https://github.com/bhawkins/hdf5_float16/blob/develop/float16.ipynb). For simplicity I used h5py, but one can obtain the same result using the HDF5 C API. HDF5 also seems to discard any payload bits in NaN values. I suspect the slowdown is due to the very general implementation for custom float types in HDF5 here

https://github.com/HDFGroup/hdf5/blob/306db409d44cccbeaff1cd5acb1a99173ac8b185/src/H5Tconv.c#L4267-L4271

versus the float16-specific handling in numpy.

The case I really care about involves a structured data type (for complex values), which is 44x slower than a numpy workaround. That demo is available [here](https://github.com/bhawkins/hdf5_float16/blob/develop/complex_float16.ipynb), though I haven't isolated a cause for that extra factor of 3x.

It seems like ideally there'd be a `H5T__conv_half_single` routine that uses hardware to convert from `_Float16` ([example](https://godbolt.org/z/M5x7oTYq8)). I guess this might require adding a `native_half` type, which seems like a big job. Or maybe just a special case in `H5T__conv_f_f`?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.