openPMD / openPMD/openPMD-api

BP4 silently drops attribute-only iterations written through the steps API

Open
#1,912 1 comment 0 reactions 0 assignees View on GitHub
affects latest release backend: ADIOS2 bug
Dominant language
C++
Stars
161
Forks
59
Avg merge
2d 22h
Merged PRs (30d)
7

Description

Verified in 0.17.1 on BP4, group- and variable-based alike: only the first attribute-only iteration survives, and neither adios2.use_group_table nor empty/constant records anchor the step.
We should probably error out if variable-based encoding is selected for BP4?

Reproducer:
```py
#!/usr/bin/env python3
# ADIOS2 BP4: attribute-only iterations are dropped when written via I/O steps.
# Verified with openPMD-api 0.16.1 and 0.17.1 (ADIOS2 2.12.1).
# HDF5 and BP5 (in all encodings) are unaffected.
# Writing via random access (series.iterations[i])
# instead of write_iterations() also preserves all iterations on BP4.
import openpmd_api as io

for ext in ("bp4", "h5"):
series = io.Series(f"attr_only.{ext}", io.Access.create)
for i in (1, 3):
it = series.write_iterations()[i]
it.set_attribute("foo", 1.0 * i) # attributes only, no datasets
it.close()
series.close()

read = io.Series(f"attr_only.{ext}", io.Access.read_only)
print(f"{ext}: iterations found = {list(read.iterations)} (expected [1, 3])")
read.close()
```
Verified output in your env (openPMD-api 0.17.1, ADIOS2 2.12.1, group-based encoding):
```
bp4: iterations found = [1] (expected [1, 3])
h5: iterations found = [1, 3] (expected [1, 3])
```

- Only the first attribute-only iteration survives; iteration 3's attributes are absent from the BP4
file entirely (confirmed with bpls -A: only /data/1/... paths exist).
- Same behavior with variable-based encoding on BP4 (read via READ_LINEAR); BP5 is unaffected in
every encoding, consistent with BP5 scoping attributes per step while BP4 writes them once into
global metadata.
- Workarounds that do not help: {"adios2": {"use_group_table": true}}, make_empty() records, and
zero-extent constant records — attributes never anchor a BP4 step.
- Workaround that does: writing through the random-access API (series.iterations[i] +
iteration.close()) instead of write_iterations() preserves all iterations on BP4 — that's what
ImpactX now does for its moments-only monitor output.

Contributor guide

Open the contributing guide

Research direction

Run the supplied Python reproducer with BP4, BP5, and HDF5, then inspect the steps API entry points for write_iterations(), random-access series.iterations[i], and iteration.close(). Compare metadata with bpls -A. Done means BP4 preserves all attribute-only iterations or clearly rejects the unsupported variable-based path, with coverage for the reproducer behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.