HDFGroup / HDFGroup/hdf5

Support data transformation w/ parallel compression

Open
#886 0 comments 0 reactions 1 assignee Claimed by @fortnern View on GitHub
Component - C Library Component - Parallel
Dominant language
C
Stars
988
Forks
355
Avg merge
4d 2h
Merged PRs (30d)
12

Description

This scenario is currently not supported, and creates an error such as this:
```
Error: #005: H5Dio.c line 1170 in H5D__ioinfo_adjust(): Can’t perform independent write with filters in pipeline.
The following caused a break from collective I/O:
Local causes: data transforms needed to be applied
Global causes: data transforms needed to be applied
major: Low-level I/O
minor: Can’t perform independent IO
Error: #004: H5Dio.c line 757 in H5D__write(): unable to adjust I/O info for parallel I/O
major: Dataset
minor: Unable to initialize object
Error: #003: H5VLnative_dataset.c line 207 in H5VL__native_dataset_write(): can’t write data
major: Dataset
minor: Write failed
Error: #002: H5VLcallback.c line 2080 in H5VL__dataset_write(): dataset write failed
major: Virtual Object Layer
minor: Write failed
Error: #001: H5VLcallback.c line 2113 in H5VL_dataset_write(): dataset write failed
major: Virtual Object Layer
minor: Write failed
Error: #000: H5Dio.c line 291 in H5Dwrite(): can’t write data
major: Dataset
minor: Write failed
```
It's difficult to understand why and how a data transformation would complicate the current implementation. See [this thread](https://forum.hdfgroup.org/t/phdf5-1-12-data-transform-in-combination-with-compression-filter/8799) on the forum.

This issue was raised by Jan-Willem Blokland.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.