DiamondLightSource / DiamondLightSource/nexgen

Use Bitshuffle instead of hdf5plugin

Open
#49 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
2
Forks
9
PR merge metrics
No merged PRs in 30d

Description

[Bitshuffle](https://github.com/kiyo-masui/bitshuffle) is generally faster than hdf5plugin for data compression.

When trying to copy and compress mask and flatfield for Tristan, there was an issue with Bishuffle that caused an `Illegal instruction (core dumped)` error when running code blocks like the following:

```
flatfield=h5py.File(flatfieldfile, "r")["image"]
bitshuffle.h5.create_bitshuffle_compressed_dataset(nxdetector, b"flatfield", shape=flatfield.shape, dtype=flatfield.dtype, chunks=(100, flatfield.shape[1]))
nxdetector["flatfield"][:] = flatfield[()]
```
```
block_size=0
flatfield=h5py.File(flatfieldfile, "r")["image"]
nxdetector.create_dataset("flatfield", data=flatfield[()], chunks=(100, flatfield.shape[1]), compression=bitshuffle.h5.H5FILTER, compression_opts=(block_size, bitshuffle.h5.H5_COMPRESS_LZ4))
```

For now nexgen is using hdf5plugin.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.