galaxyproject / galaxyproject/idc

Fixes for idempotency

Open
#30 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Shell
Stars
10
Forks
8
PR merge metrics
No merged PRs in 30d

Description

Right now, the determination of what needs to be done is keyed on the existence of a history matching `-` under the `idc` user on usegalaxy.org. But this has multiple problems:

1. If a DM fails, it will not be retried unless the history is manually deleted
2. If the bundle export/import fails, it will never be retried
3. If the CVMFS publish fails, it will never be retried
4. If building is successful and you just want to retry export/import or publish, this is not possible, you have to delete the histor(ies) and retry from the beginning.

In addition to the existence of the history, we should check for a successful DM execution in that history - or, we should make sure the CI script always deletes histories when a DM run fails. @jmchilton since you did the Ephemeris part of this do you have thoughts or can you help out here?

We need to check what's actually published on CVMFS in order to determine what needs to be exported/imported and published. I can do this part of it. The only issue is that I was publishing the `run_data_managers.yaml` files into CVMFS as a sort of "receipt" of what was done, but this file will not be generated by the previous script if the history already exists. I thought about maybe uploading it to the history before the DM run. We could also just regenerate it as needed, or stop publishing it altogether.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by locating the CI script and the flow that runs Data Managers, exports/imports histories, and publishes to CVMFS. Review how run_data_managers.yaml is generated and used as a receipt. Done means failed stages can be retried independently and the workflow determines completion from successful execution and actual CVMFS contents.

Written by the indexing model from the issue text.

Assessment

Tech stack
shell
Domain
ci-cd, devops, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.