HDFGroup / HDFGroup/hdf5

Document best practices and available controls for testing of HDF5

Open
#6,120 0 comments 0 reactions 1 assignee Claimed by @jhendersonHDF View on GitHub
Component - Documentation
Dominant language
C
Stars
988
Forks
355
Avg merge
4d 2h
Merged PRs (30d)
12

Description

HDF5 should document known best practices and available knobs for controlling how and where HDF5 can be tested when building the library.

A relevant comment from @gheber on https://github.com/HDFGroup/hdf5/issues/6114, in the context of test failures occurring due to running HDF5 testing on an NFS mount:

> Maybe we should give people a short guide like this:
>
> **Key knobs for keeping builds/tests off slow filesystems**
>
> - CMAKE_RUNTIME_OUTPUT_DIRECTORY, CMAKE_LIBRARY_OUTPUT_DIRECTORY, CMAKE_ARCHIVE_OUTPUT_DIRECTORY,
> CMAKE_TEST_OUTPUT_DIRECTORY, CMAKE_Fortran_MODULE_DIRECTORY: defaults point to /bin (and /mod for modules) but can be set at configure time to a fast scratch path, e.g., cmake -DCMAKE_RUNTIME_OUTPUT_DIRECTORY=/scratch/hdf5/bin -DCMAKE_TEST_OUTPUT_DIRECTORY=/scratch/hdf5/tests ..
(config/HDFMacros.cmake:41-88). CMAKE_INSTALL_PREFIX is also set here; override it if you want the install
tree on a different volume.
> - HDF5_PREFIX: prepends a path to every serial test file; great for moving serial test I/O to a fast disk
(test/h5test.c:50-54). Example: export HDF5_PREFIX=/scratch/$USER/hdf5_serial.
> - HDF5_PARAPREFIX: same idea for MPI/parallel tests; critical to steer MPI-IO away from NFS/home (test/
h5test.c:56-61,385-415; release_docs/README_HPC.md:302-334). Example: export HDF5_PARAPREFIX=/scratch/
$USER/hdf5_par.
> - HDF5_DRIVER / HDF5_TEST_DRIVER: selects the VFD for tests (test/h5test.c:42-45,2113). Setting
HDF5_DRIVER=core keeps test files in memory (with an optional backing store), which can avoid slow storage
timeouts; other values like sec2, split, multi, and family change how files are laid out.
> - HDF5_NOCLEANUP: when set, test files are left on disk (useful for debugging, but can fill slow disks)
(test/h5test.c:370-383).
> - Test runtime controls to reduce exposure to slow disks: HDF5TestExpress (or the doc alias
HDF5_TEST_EXPRESS) picks quick vs exhaustive suites (test/h5test.c:910-923; release_docs/README_HPC.md:335-
343), and -DDART_TESTING_TIMEOUT=... raises the default 1200s test timeout if a slow FS is unavoidable
(release_docs/README_HPC.md:344-350).
>
> Suggested use on slow filesystems: configure with runtime/archive/test output dirs on a fast scratch volume;
export HDF5_PREFIX/HDF5_PARAPREFIX accordingly; optionally use HDF5_DRIVER=core for tests and a higher
DART_TESTING_TIMEOUT or lower express level if still hitting timeouts.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.