HDFGroup / HDFGroup/hdf5

Performance Issue: Excessive Function Calls for Chunked Dataset Reads with Non-Contiguous Memory Layout

Open
#6,058 0 comments 0 reactions 1 assignee Claimed by @jhendersonHDF View on GitHub
Component - C Library HDFG-internal
Dominant language
C
Stars
988
Forks
355
Avg merge
4d 2h
Merged PRs (30d)
12

Description

## Description

When reading chunked datasets where the chunk shape doesn't match the memory buffer layout, the HDF5 library makes an excessive number of small read operations, resulting in severe performance degradation.

## Environment

- Dataset dimensions: 35,660,170 × 64
- Chunk size varies during experiments
- Platform: Linux compute node
- Single node execution

## Problem Details

When chunking is enabled, the following functions are called millions of times:
- `H5F_shared_block_read`
- `H5F__accum_read`

### Specific Example

Reading a single hyperslab (10000 × 8) resulted in:
- **79,198 calls** to `H5D__contig_readvv_sieve_cb`
- Issue is more prone to occur when:
- Size is not divisible by chunk size
- File is chunked in both dimensions

## Steps to Reproduce

```c
// Straightforward write and read operation
// Dataset: 35,660,170 × 64
// Chunk the dataset in the first dimension (problematic)
// vs. chunking in the second dimension (better performance)
// Read hyperslab: 10000 × 8
```

See attached test code (write and read operations mimicking array management library performance experiments).

## Root Cause Analysis

The library performs naive iteration over the memory selection in chunks to generate offset/length pairs for copying from file to memory buffer. Key findings:

1. **Serial Case**: When chunk shape doesn't match memory buffer layout, the library iterates element-by-element
2. **Element-wise reads**: Each call to `H5D__contig_readvv` reads a 4-byte element in the second dimension according to chunk shape, then proceeds to next element in first dimension
3. **Result**: Huge number of I/O calls and excessive file seeking
4. **Sieve buffer**: Optimizes slightly (reading 4KiB worth of elements per iteration), but limited effectiveness

### Call Analysis

- `H5F_shared_block_read` goes almost directly to the underlying VFD and then to the file
- The sieve buffer (likely 64KiB) provides some mitigation
- Without special settings, the issue stems from the sheer number of calls generated by the library

## Performance Impact

### Default Behavior (Problematic)
- Read time: **1.5 seconds**
- Very high number of function calls

### With Chunk Cache (Workaround)
- Increase chunk cache to at least 1 chunk size (≥142,640,680 bytes)
- Read time: **19 microseconds** (significant improvement)
- Uses 128 `H5D__compact_readvv` calls
- Still **13 microseconds slower** than optimal case (6 microseconds)
- Scaling: For larger datasets, optimal case takes 0.12s vs cached solution 0.54s

### With MPI I/O Driver + Collective I/O
- Read time: **21 microseconds**
- No anomaly in number of read calls
- Increased malloc calls: 2,743 → 38,627 (sum duration: 6μs → 100μs)
- Selection I/O successfully used (64 dataspaces for 64 chunks → single MPI I/O call)
- **Note**: Both OpenMPI and MPICH allocate much larger memory than expected

### With MPI I/O Driver + Independent I/O
- Read time: **8.6 seconds** (7.7s in H5Dread)
- **15,668 calls** to `H5D__contig_readvv_cb` (worse than serial case: 14,173)
- **129k malloc calls** (sum duration: 2 seconds) - 4x more than collective
- Translates to many individual 4-byte reads, similar to serial case
- Derived datatype composed of all 4-byte reads causes MPI to struggle with large memory overhead

### Optimal Case (Chunked in Second Dimension)
- Read time: **6 microseconds**
- Follows simple procedure with fewer function calls
- Significantly outperforms both cached and collective approaches

## Testing with MPI I/O

Using MPI I/O driver (even with single process):
- Total events captured: **halved**
- **1.5 million** `PMPI_File_read_at` calls
- **1 million** `MPI_File_read_at` calls
- Selection I/O appears to be getting skipped (expected only one `MPI_File_read_at` per `H5Dread`)

### Selection I/O Diagnostic

Can check why selection I/O is disabled using:
```c
H5Pget_no_selection_io_cause()
```
Returns bitfield of reasons for disabling selection I/O.

## Additional Observations

### I/O Sieve Buffer
- Currently **not being used** at all (likely due to recent refactoring of dataset layout-specific code)
- Force-enabling I/O sieve buffer gives much better results
- Still slower than dataset chunk cache case due to current library architecture

### Independent I/O File Close
- `H5Fclose` in independent mode uses ~2.5x more barriers than collective mode
- Suggests optimization opportunities in finalization phase

## Proposed Solutions

### Short-term Workaround
**Applications should increase chunk cache size to at least 1 chunk** when dataset's chunking shape isn't contiguous with memory.

### Long-term Optimizations Needed

1. **Higher-level library optimization**
- Minimize number of offset/length pairs generated
- Two-phase approach: read with chunk shape (64 I/O calls at worst), then perform memory scatter
- Consider using preadv() in sec2 VFD to let OS handle inefficiencies

2. **VFD-level optimization**
- Use POSIX "v" I/O functions (preadv/pwritev)
- Let OS optimize when library generates suboptimal patterns

3. **General refactoring**
- Chunked dataset I/O needs architectural improvements
- Re-enable and optimize I/O sieve buffer usage

4. **MPI Independent I/O**
- Currently no optimization (though theoretically could optimize for single process case)
- Needs additional MPI support for better optimization

## Files and Call Graphs

See attached:
- Test code for reproduction (write and read operations)
- Call graphs for various scenarios
- Performance analysis document

## Notes

- Selection I/O with default sec2 VFD currently doesn't implement the needed optimizations
- Collective I/O often receives focus; independent I/O needs more attention
- The naive iteration behavior shifts from high-level I/O to lower-level when chunk cache is used, but doesn't eliminate the problem

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.