openPMD / openPMD/openPMD-api

Overhead with variable-based `bp5` series

Open
#1,859 18 comments 1 reaction 3 assignees Claimed by @guj View on GitHub
affects latest release backend: ADIOS2 bug performance
Dominant language
C++
Stars
161
Forks
59
Avg merge
2d 22h
Merged PRs (30d)
7

Description

**Describe the bug**
Performance kill when opening/closing a variable-based `bp5` series with many iterations.

**To Reproduce**
Standalone reproducer in Python:

```python
import openpmd_api as io
from tqdm import tqdm
adios2_cfg = r'''
{
"iteration_encoding": "variable_based",
"adios2": {
"engine": {
"type": "bp5",
"parameters": {
"StatsLevel": "0"
}
}
}
}
'''
OUTPUT_PATH = "myseries.bp5"
series = io.Series(OUTPUT_PATH, io.Access_Type.create_linear, adios2_cfg)
N = 1000000
for i in tqdm(range(N)):
it = series.snapshots()[i]
it.close()
```

**Expected behavior**
I expect much less overhead just to open/close the iterations, and not to change in time.

**Software Environment**
- version of openPMD-api: 0.17.0
- installed openPMD-api via: conda-forge
- operating system:
- machine: Perlmutter at NERSC - login nodes
- name and version of Python implementation: CPython 3.11.14
- version of ADIOS2: 2.11.0
- name and version of MPI: N/A

**Additional context**
The overhead builds up as the loop moves through the iterations.
Starts around ~1k iterations/s and goes down to ~200 iterations/s after ~100k iterations.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.