openPMD / openPMD/openPMD-api

ADIOS2: Flush Attributes Earlier

Open
#984 1 comment 1 reaction 0 assignees View on GitHub
backend: ADIOS2 feature request
Dominant language
C++
Stars
161
Forks
59
Avg merge
2d 22h
Merged PRs (30d)
7

Description

@RemiLehe and @Yin-YinjianZhao are running WarpX backtransformed diagnostics (cc @RevathiJambunathan) with ADIOS2.

The characteristics of this I/O methods is that we open an iteration a very long time, usually most of the runtime of a simulation, and add more-and-more `storeChunk`s to the written data.

With HDF5, they can follow the data as it is written and "just" have some undefined/invalid values in the file as the data is not yet complete. With ADIOS2, the same is in principle possible. (General note: both in HDF5 & ADIOS2, unwritten data is _undefined_, which is just by coincidence often zero. An analysis on the data should skip the unwritten parts.)

The problem with ADIOS2 (tested with fileBased output) right now is though, that we are not writing the attributes out until the file is closed (i.e. `/openPMD`). In practice, we declare all attributes early enough though, so that we could indicate/promise from the user-side at some point forward that:
- no further attribute changes and
- no new declarations of variables will be coming and
- that we only do further `storeChunk` calls before the pending close.

In that case, we should be able to flush the attributes when the user gives us a hint... What do you think @franzpoeschel?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.