HDFGroup / HDFGroup/hermes

Feature Request: Persistent file storage from Hermes

Open
#683 2 comments 2 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
34
Forks
19
PR merge metrics
No merged PRs in 30d

Description

### Requested feature:
Persistent file storage from Hermes

### Problem:

We are using Hermes as the IO engine for ADIOS2 in scientific workflows.
Whenever our workflow finishes, Hermes has no means to store the collected data to disk and therefore the data is lost.
As the raw data from the workflow is not saved it is not possible to perform post-hoc analysis.

As these workflows can be computationally expensive it is not practical to re-run experiments to regenerate the data.
Further, if the workflow is of a stochastic or chaotic nature, it may not be possible to replicate previous runs.

### Proposed solution:

When Hermes is preparing to clear data, add an option so that this data can be written to disk.

For example, the default engine for ADIOS2 is the bp5 engine.
Hermes could be configured to write to disk data that it no longer is using with the bp5 format.
At the end of the workflow when Hermes exits, an end user would have access to the raw data in bp5 format.
This way the user gets the benefit of using the Hermes engine while in the workflow but also can access the results of the experiment after it was run.

Contributor guide

No contributing guide indexed for this repository

Research direction

The issue names Hermes, ADIOS2, and the bp5 format but does not identify source files, tests, or an entry point. Start by locating the data-clearing and shutdown paths, then clarify the persistence option, output lifecycle, and acceptance tests needed to show that collected data is available in bp5 format after the workflow ends.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.