Segmentation fault when writing large files collectively with Fortran API
- Dominant language
- C
- Stars
- 988
- Forks
- 355
- Avg merge
- 4d 2h
- Merged PRs (30d)
- 12
Description
**Describe the bug**
When writing large files collectively from more then 3 Processes, the program ends with a segmentation fault while writing. For gcc builds on various systems. Large means about >=8 GB per process. We call the HDF5 Fortran API.
I include a minimal failing example. Each process allocates an int32 one-rank array of size 2147483647 (that are ~8.59GB) initialized with the process number and writes that to a one-rank dataset of size `n_processes * 2147483647` of type `H5T_NATIVE_INTEGER`. The program ends in the segmentation fault when called from more then three processes/
**Expected behavior**
The program writes the expected file collectively and ends without an error.
**Platform**
The example code fails as described on two systems:
- Laptop:
- Dell Latitude 5430
- 12th Gen Intel(R) Core(TM) i7-1265U
- 32 GB RAM
- HPC:
- Dell R7525 Server, 2x 7713 AMD EPYC with 1TB RAM with 2x 100 GBit/s Ethernet
- Rocky 9.3 or 9.4 as hypervisor
- Runs a KVM in a docker container
- Connected to a Lustre with 100GB/s Ethernet, 2 meta and 4 storage servers
- HDF5 version
- 1.12.3 and 1.14.0 (on both systems)
- OS and version
- Ubuntu 24.04.1 LTS (Laptop)
- OpenSUSE 15.5 (HPC)
- Compiler and version
- gcc 13.3.0 (Laptop)
- gcc 14.2.0 (HPC)
- Build system (e.g. CMake, Autotools) and version
- Any configure options you specified
- `export CC=mpicc F9X=mpif90 ./configure --enable-build-mode=production --enable-parallel --enable-fortran` (Laptop and HPC for HDF5-1.12.3)
- `export CC=mpicc CXX=mpicxx FC=mpifort F77=mpif77 ; ./configure --prefix=/software/hdf5/1.14.5-libaec-1.1.3-openmpi-5.0.6-gcc14.2.0 --with-pic --with-pthread --enable-shared --enable-threadsafe --enable-unsupported --enable-parallel --enable-fortran --with-fortran2003 --enable-build-mode=production` (HPC for HDF5-1.14.0)`
- MPI library and version (parallel HDF5)
- OpenMPI 4.1.6 (Laptop)
- OpenMPI 5.0.6 (HPC)
**Additional context**
The program ends without an error if the `h5dwrite_f` routines is commented out or if, in the `h5pset_dxpl_mpio_f` call, `H5FD_MPIO_COLLECTIVE_F` is replaced with `H5FD_MPIO_INDEPENDENT_F`.
The program runs as expected on the LUMI HPC in Finland with cray-HDF5-12.2.11 built with gcc 12.2.0 and on intel nodes (Xeon(R) Gold 6126 CPU @ 2.60GHz) built with intel-oneapi 2021.4.0. On the intel machine, HDF5-1.12.0 is self built with `export CC=mpiicc F9X=mpiifort .configure --enable-build-mode=production --enable-parallel --enable-fortran --enable-fortran2003`
Additionally I tested on the Laptop HDF5 installed with `apt-get install libhdf5-openmpi-dev` which is version 1.10.10. In this case the program fails as well as described above.
All software on the HPC is self built. Here are the specs:
GCC 14.2.0
```./contrib/download_prerequisites``` gefolgt von ```./configure --prefix=/software/gcc/14.2.0 --disable-multilib```
libaec 1.1.3
```cmake -S ../ -DCMAKE_INSTALL_PREFIX=/software/libaec/1.1.3 ; make ; make install```
OpenMPI 5.0.6 gcc/14.2.0
``` ./configure --prefix=/software/openmpi/5.0.6-gcc14.2.0 ; make -j ; make install```
HDF5 1.14.5 libaec/1.1.3 openmpi/5.0.6 gcc/14.2.0
```export CC=mpicc CXX=mpicxx FC=mpifort F77=mpif77 ; ./configure --prefix=/software/hdf5/1.14.5-libaec-1.1.3-openmpi-5.0.6-gcc14.2.0 --with-pic --with-pthread --enable-shared --enable-threadsafe --enable-unsupported --enable-parallel --enable-fortran --with-fortran2003 --enable-build-mode=production```
**Failing Example `parallel_write_int32.f90`**
```Fortran
program parallel_write_int32
use iso_fortran_env
use iso_c_binding
use hdf5
use mpi
implicit none
character(*), parameter :: file_name = "test_file.h5"
character(*), parameter :: dataset = "dataset"
integer(int64), parameter :: block_size=2147483647
integer :: ierr, procs, rank
integer(HID_T) :: file_id, file_id_plist, dataspace_id, dataset_id, memspace_id
integer(HSIZE_T) :: dims(1), offset(1), datacount(1)
integer(int32), allocatable, target :: data_block(:)
type(c_ptr) :: buffer
! Initialize MPI
call mpi_init(ierr)
call mpi_comm_size(mpi_comm_world, procs, ierr)
call mpi_comm_rank(mpi_comm_world, rank, ierr)
! Prepare Data
dims = [int(block_size, HSIZE_T) * procs]
datacount = [int(block_size, HSIZE_T)]
offset = [int(rank, HSIZE_T) * block_size] ! HDF5 assumes c-like indexing
allocate(data_block(block_size), source=rank)
buffer = c_loc(data_block)
! Initialize HDF5 file
call h5open_f(ierr)
! Create file and open root group
call h5pcreate_f(H5P_FILE_ACCESS_F, file_id_plist, ierr)
call h5pset_fapl_mpio_f(file_id_plist, mpi_comm_world, MPI_INFO_NULL, ierr)
call h5fcreate_f(file_name, H5F_ACC_TRUNC_F, file_id, ierr, access_prp = file_id_plist)
call h5pclose_f(file_id_plist, ierr)
! Create dataset for the global array
call h5screate_simple_f(size(dims), dims, dataspace_id, ierr)
call h5dcreate_f(file_id, dataset, H5T_NATIVE_INTEGER, dataspace_id, dataset_id, ierr)
call h5sclose_f(dataspace_id, ierr)
! Prepare dataspace for writing
call h5screate_simple_f(size(dims), datacount, memspace_id, ierr)
call h5dget_space_f(dataset_id, dataspace_id, ierr)
call h5sselect_hyperslab_f(dataspace_id, H5S_SELECT_SET_F, offset, datacount, ierr)
! Create a property list to enable parallel writing
call h5pcreate_f(H5P_DATASET_XFER_F, file_id_plist, ierr)
call h5pset_dxpl_mpio_f(file_id_plist, H5FD_MPIO_COLLECTIVE_F, ierr) ! H5FD_MPIO_INDEPENDENT_F works
! Write data to file
call h5dwrite_f(dataset_id, H5T_NATIVE_INTEGER, buffer, ierr, &
file_space_id = dataspace_id, mem_space_id = memspace_id, xfer_prp = file_id_plist)
! Close open HDF5 objects
call h5pclose_f(file_id_plist, ierr)
call h5sclose_f(dataspace_id, ierr)
call h5sclose_f(memspace_id, ierr)
call h5dclose_f(dataset_id, ierr)
call h5fclose_f(file_id, ierr)
call h5close_f(ierr)
! Finalize MPI
call mpi_finalize(ierr)
end program
```
Built with
```bash
h5pfc parallel_write_int32.f90 -g -O0 -fbounds-check -fbacktrace -Wall -Wextra -o parallel_write_int32
h5pfc parallel_write_int32.f90 -g -O3 -o parallel_write_int32
```
Contributor guide
Assessment
This issue has not been assessed yet.