CROCODILE-CESM / CROCODILE-CESM/workshop25-hack-projects

CESM Hybrid multi-instance run

Open
#4 0 comments 0 reactions 0 assignees View on GitHub
bug CESM CIME Python
Dominant language
No language data
Stars
0
Forks
0
PR merge metrics
No merged PRs in 30d

Description

## CESM / MOM_interface — Hybrid Multi-Instance Restart Precheck Fix

**Summary**
CESM can be initialized in one of three ways: startup, hybrid and branch.
https://ncar.github.io/CESM-Tutorial/notebooks/modifications/xml/modify_run_type/hybrid_branch_restart.html

Hybrid is a common way to use and existing ensemble of CESM in a data assimilation experiment. However, currently the setup for hybrid for the MOM6 interface for CESM is incorrect.

This project aims to correct the logic in the MOM_interface’s `cime_config/buildnml` module that verifies the presence of restart files so that it correctly handles *multi-instance* cases.
Currently, the precheck only looks for a single-instance restart file (e.g.,
`${CASE}.mom6.r.${DATE}.nc`), and fails when instance-specific filenames (e.g.,
`${CASE}.mom6_${INST}.r.${DATE}.nc` or `${CASE}.mom6.r.${DATE}._${INST}.nc`) should instead be validated.
See [Issue #275](https://github.com/ESCOMP/MOM_interface/issues/275).

---

### Motivation
- The current bug prevents proper checking of restart files in multi-instance (ensemble) runs, leading to errors when users setup the case
- Hybrid or ensemble CESM / MOM workflows often rely on multiple instances of the ocean model (MOM6), so this precheck is a critical component for correct initialization.

---

### Goals
* Analyze the existing `buildnml` precheck logic (around lines 403–405).
*. Add or restructure conditional logic to detect restart files in multi-instance naming conventions (old and new formats), e.g.:
- `${CASE}.mom6.r.${DATE}._${INST}.nc`
- `${CASE}.mom6_${INST}.r.${DATE}.nc`
For restarts, thickness files, and any other instance specific files.
* Implement regression or unit tests to verify correct behavior for both single- and multi-instance runs.

---

### Approach / Methodology
- **Inspect and refactor:** Review the restart-check logic in `buildnml` and identify where it assumes single-instance behavior.
- **Add multi-instance support:** When `NINST_OCN > 1` (or similar), loop over instance indices and check for multiple naming patterns.
- **Provide error handling:** Report clearly if instance(s) are missing restart files.
- **Testing and validation:** Run both single- and multi-instance cases to confirm correct file detection and failure handling.

---

### Skills Needed or To be Gained with this Project
- Familiarity with **CESM / CIME / MOM_interface** build and run systems.
- Proficiency in **Python**, **shell scripting**, and **Git**.
- Understanding of **multi-instance / ensemble modeling** concepts.
- Ability to interpret CESM restart and configuration files.

---

### Possible Challenges & Open Questions
- Handling both single and new restart naming conventions consistently.
- Interactions with `continue_run`.
- Multi instance branch mode may also have the same bug.

---

### References
- [Issue #275 — “precheck test of restart existence doesn’t handle multi-instance”](https://github.com/ESCOMP/MOM_interface/issues/275)
- [MOM_interface source code (buildnml)](https://github.com/ESCOMP/MOM_interface/)
- [CESM / CIME User Guide](https://esmci.github.io/cime/versions/master/html/users_guide/)
- CESM and MOM6 multi-instance documentation

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.