DiamondLightSource / DiamondLightSource/nexgen
Use Bitshuffle instead of hdf5plugin
- Dominant language
- Python
- Stars
- 2
- Forks
- 9
- PR merge metrics
- No merged PRs in 30d
Description
[Bitshuffle](https://github.com/kiyo-masui/bitshuffle) is generally faster than hdf5plugin for data compression.
When trying to copy and compress mask and flatfield for Tristan, there was an issue with Bishuffle that caused an `Illegal instruction (core dumped)` error when running code blocks like the following:
```
flatfield=h5py.File(flatfieldfile, "r")["image"]
bitshuffle.h5.create_bitshuffle_compressed_dataset(nxdetector, b"flatfield", shape=flatfield.shape, dtype=flatfield.dtype, chunks=(100, flatfield.shape[1]))
nxdetector["flatfield"][:] = flatfield[()]
```
```
block_size=0
flatfield=h5py.File(flatfieldfile, "r")["image"]
nxdetector.create_dataset("flatfield", data=flatfield[()], chunks=(100, flatfield.shape[1]), compression=bitshuffle.h5.H5FILTER, compression_opts=(block_size, bitshuffle.h5.H5_COMPRESS_LZ4))
```
For now nexgen is using hdf5plugin.
Contributor guide
Assessment
This issue has not been assessed yet.