openPMD / openPMD/openPMD-api

Crash with `HDF5` backend due to incompatible `MPI_File_Sync` policy

Open
#1,423 12 comments 0 reactions 2 assignees Claimed by @jeanbez View on GitHub
backend: HDF5 MPI question
Dominant language
C++
Stars
161
Forks
59
Avg merge
2d 22h
Merged PRs (30d)
7

Description

Writing a dataset using `OpenPMD` on an `NFS` filesystem at Fermilab's wilson-cluster, [work1](https://computing.fnal.gov/wilsoncluster/filesystems/) as part of a simple `fodo_cxx` simulation fails:
```
HDF5-DIAG: Error detected in HDF5 (1.14.0) MPI-process 0:
#000: /tmp/sasyed/spack-stage/spack-stage-hdf5-1.14.0-fs2ybrgllsgpb3ltdwkbqv5a3seilqya/spack-src/src/H5D.c line 1390 in H5Dwrite(): can't synchronously write data
major: Dataset
minor: Write failed
#001: /tmp/sasyed/spack-stage/spack-stage-hdf5-1.14.0-fs2ybrgllsgpb3ltdwkbqv5a3seilqya/spack-src/src/H5D.c line 1333 in H5D__write_api_common(): can't write data
major: Dataset
minor: Write failed
#002: /tmp/sasyed/spack-stage/spack-stage-hdf5-1.14.0-fs2ybrgllsgpb3ltdwkbqv5a3seilqya/spack-src/src/H5VLcallback.c line 2282 in H5VL_dataset_write_direct(): dataset write failed
major: Virtual Object Layer
minor: Write failed
#003: /tmp/sasyed/spack-stage/spack-stage-hdf5-1.14.0-fs2ybrgllsgpb3ltdwkbqv5a3seilqya/spack-src/src/H5VLcallback.c line 2237 in H5VL__dataset_write(): dataset write failed
major: Virtual Object Layer
minor: Write failed
#004: /tmp/sasyed/spack-stage/spack-stage-hdf5-1.14.0-fs2ybrgllsgpb3ltdwkbqv5a3seilqya/spack-src/src/H5VLnative_dataset.c line 408 in H5VL__native_dataset_write(): can't write data
major: Dataset
minor: Write failed
#005: /tmp/sasyed/spack-stage/spack-stage-hdf5-1.14.0-fs2ybrgllsgpb3ltdwkbqv5a3seilqya/spack-src/src/H5Dio.c line 673 in H5D__write(): unable to adjust I/O info for parallel I/O
major: Dataset
minor: Unable to initialize object
#006: /tmp/sasyed/spack-stage/spack-stage-hdf5-1.14.0-fs2ybrgllsgpb3ltdwkbqv5a3seilqya/spack-src/src/H5Dio.c line 1211 in H5D__ioinfo_adjust(): Can't perform independent write when MPI_File_sync is required by ROMIO driver.
major: Dataset
minor: Can't perform independent IO
[AbstractIOHandlerImpl] IO Task WRITE_DATASET failed with exception. Clearing IO queue and passing on the exception.
[HDF5] Internal error: Failed to write dataset /data/0/particles/track_coords/moments/x
Abort(888) on node 0 (rank 0 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 888) - process 0
srun: error: wcgwn013: task 0: Exited with exit code 120
```

First observed by @egstern.

**Software Environment**
- version of openPMD-api: `0.15.1`
- installed openPMD-api via: `spack`
- operating system: `centos-7`
- machine: `wilson-cluster@fermilab`
- name and version of Python implementation: `python-3.11`
- version of HDF5: `1.14.0`
- version of ADIOS2: `2.8.3`
- name and version of MPI: `MPICH-4.1.1`

**Spack spec**
Full spec is here: https://gist.github.com/s-sajid-ali/0596b18b83400172067c417932b1a852

**Additional Information**
Let me know if I should try building `openpmd-api`with testing enabled (maybe add a spack variant for it) and see if that succeeds?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.