Make "compare" task derive METRIC_FILES dynamically from recipe datasets (multi-run safe)
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 5
- Forks
- 3
- Avg merge
- 5d 6h
- Merged PRs (30d)
- 8
Description
**Description**
The current `compare `task relies on hard-coded **METRIC_FILES** in
`app/compare/opt/rose-app-.conf,` e.g.:
```
HadGEM3-GC31-LL_metrics.csv
UKESM1-0-LL_metrics.csv
```
This approach breaks as soon as:
• the evaluation model changes,
• additional runs are introduced (e.g. eval2, eval3),
• or dataset names are modified via the recipe.
**Observed Errors**
During recent test runs, the `compare `task failed with:
`... UKESM1-0-LL_metrics.csv` does not exist
Even though:
• the file did exist for the actual dataset used, and
• other metric files in the same loop expanded correctly.
Root cause:
• METRIC_FILES is model-name hard-coded and no longer matches the
datasets produced by `run_recipe`.
• Shell glob expansion masked the issue for the first file but failed for subsequent entries.
Root Cause Summary
• `compare.sh` assumes exact model identifiers in filenames.
• These identifiers are duplicated in Rose config instead of being derived from the recipe.
• This creates technical debt and blocks multi-run scalability.
________________________________________
**Proposed Solution**
Derive METRIC_FILES dynamically from the ESMValTool recipe at runtime.
Implementation outline:
1. Parse the datasets list from RECIPE_PATH in `compare.sh `(or a helper script).
2. Extract dataset names actually used by the recipe.
3. Construct expected metric and plot filenames programmatically.
4. Validate existence of generated paths instead of static strings.
**Benefits**:
• Fully multi-run compatible (ref, eval, eval2, …).
• Eliminates hard-coded model names in Rose configuration.
• Ensures compare logic stays consistent with the recipe used to generate outputs.
• Reduces maintenance and configuration duplication.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with app/compare/compare.sh and its RECIPE_PATH handling, then inspect app/compare/opt/rose-app-.conf and the outputs from run_recipe. Done means metric and plot paths are derived from datasets in the active recipe, generated paths are validated, and compare works for ref, eval, and additional runs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- shell
- Domain
- devops
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100