openPMD / openPMD/openPMD-api

Close for Record and RecordComponent

Open
#778 0 comments 0 reactions 0 assignees View on GitHub
feature request frontend: C++17 frontend: Python3 internal MPI
Dominant language
C++
Stars
161
Forks
59
Avg merge
2d 22h
Merged PRs (30d)
7

Description

In AMR codes such as WarpX, we have the challenge that the number of `storeChunk` calls is a runtime parameter for the number of blocks of an MPI rank. This means that we cannot express those calls as collective calls.

Nevertheless, something that we can enforce (as an API contract) is that the order of variable writing (and reading) in parallel I/O needs to be in-order. By that way, we can create "collective" regions of I/O within time steps (`Iteration::close()`) and even record and record component transactions.

Concretely, as in for `Iteration::close()` in #746, we should add MPI-collective member functions:
- `RecordComponent::close()`
- `Record::close()`

to signal no further `storeChunk` (or `loadChunk`) calls will be issued on those variables (or blocks of variables). As always (e.g. with particles), it is okay that some ranks might contribute zero blocks (no `storeChunk()`) to a variable - and we can contractually state that `::close()` still needs to be called in such a case (collective call).

Adding such methods will enable backends to perform optimizations in streaming and staging to reduce the overhead from many `storeChunk` calls.

cc @guj @franzpoeschel

Refs.: ADIOS2-WarpX meeting from Sep 16th, 2020.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.