Feature Request: Persistent file storage from Hermes
- Dominant language
- C++
- Stars
- 34
- Forks
- 19
- PR merge metrics
- No merged PRs in 30d
Description
### Requested feature:
Persistent file storage from Hermes
### Problem:
We are using Hermes as the IO engine for ADIOS2 in scientific workflows.
Whenever our workflow finishes, Hermes has no means to store the collected data to disk and therefore the data is lost.
As the raw data from the workflow is not saved it is not possible to perform post-hoc analysis.
As these workflows can be computationally expensive it is not practical to re-run experiments to regenerate the data.
Further, if the workflow is of a stochastic or chaotic nature, it may not be possible to replicate previous runs.
### Proposed solution:
When Hermes is preparing to clear data, add an option so that this data can be written to disk.
For example, the default engine for ADIOS2 is the bp5 engine.
Hermes could be configured to write to disk data that it no longer is using with the bp5 format.
At the end of the workflow when Hermes exits, an end user would have access to the raw data in bp5 format.
This way the user gets the benefit of using the Hermes engine while in the workflow but also can access the results of the experiment after it was run.
Contributor guide
No contributing guide indexed for this repository
Research direction
The issue names Hermes, ADIOS2, and the bp5 format but does not identify source files, tests, or an entry point. Start by locating the data-clearing and shutdown paths, then clarify the persistence option, output lifecycle, and acceptance tests needed to show that collected data is available in bp5 format after the workflow ends.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100