ESMCI / ESMCI/cime

Refactor Slice 2: Move batch/scheduler logic into `CIME/core/batch/`

Open
#4,997 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
174
Forks
225
Avg merge
1d 16h
Merged PRs (30d)
14

Description

Part of #4950. Slice 2 of 5. Estimated 4 weeks. Depends on #4996.

## Goal

Move batch logic from scattered locations (`XML/env_batch.py`,
`case/case_submit.py`, `utils.py`) into a coherent `CIME/core/batch/` package.
Existing entrypoints become thin wrappers.

## Why

`EnvBatch` (`XML/env_batch.py`, 1590 lines, 43 methods) is the second-largest
class in CIME. It handles batch config parsing, job submission, queue
management, and directive generation all in one class. Submission logic is
also smeared into `case_submit.py` and `utils.py`.

## Tasks

### Decompose `EnvBatch`
- [ ] `CIME/core/batch/config.py` — batch config / queue parsing.
- [ ] `CIME/core/batch/submit.py` — job submission logic (consumes from
`EnvBatch` and `case_submit.py`).
- [ ] `CIME/core/batch/directives.py` — directive generation.
- [ ] `EnvBatch` becomes a thinner wrapper delegating to the above.

### Scheduler protocol
- [ ] Define `Scheduler` protocol in `CIME/core/batch/scheduler.py`.
PBS, Slurm, LSF are genuine polymorphic backends — DI is warranted here.
- [ ] Concrete implementations: `PBSScheduler`, `SlurmScheduler`, `LSFScheduler`.

### Move from `utils.py` → `core/batch/`
- [ ] `batch_jobid`, `get_batch_script_for_job`, `get_project`,
`get_charge_account`, `add_mail_type_args`, `resolve_mail_type_args`,
`transform_vars`.

### Move from `utils.py` → `core/shell.py`
- [ ] `run_cmd`, `run_cmd_no_fail` (general utility, pulled here because batch
is the first heavy consumer).

### Consolidation
- [ ] `transform_vars` (`utils.py:2118`) takes `case` and calls
`case.get_value()` repeatedly — consolidate as a `Case` method, with
core template logic in `CIME/core/batch/`.
- [ ] `batch_jobid` (`utils.py:2360`) calls `case.get_job_id()` — move into
the batch subsystem.

### Imports & re-exports
- [ ] Update internal CIME imports to point at `core/batch/`.
- [ ] Leave re-exports in `utils.py`, `case/case_submit.py` for external
consumers.

## Definition of done

- [ ] All batch tests pass.
- [ ] External models (E3SM, CESM, NorESM) submit jobs unchanged.
- [ ] Batch logic lives in `core/batch/`, not in `XML/`, `case/`, `utils.py`.
- [ ] `EnvBatch` is a thin wrapper.
- [ ] `Scheduler` protocol with PBS/Slurm/LSF implementations.
- [ ] 80%+ coverage on `core/batch/`.

## Related

- Depends on #4996 (foundation, exception hierarchy)
- Blocks #4999 (build orchestration may depend on batch interface)

Contributor guide

Open the contributing guide

Research direction

Start by reading XML/env_batch.py, case/case_submit.py, and the listed utilities, then review dependency #4996 and the existing batch tests. Map current submission, queue, directive, and scheduler entry points before dividing the work across CIME/core/batch/. Done means all batch tests pass, external models submit unchanged, EnvBatch is a thin wrapper, and the stated coverage and backend requirements are met.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
infrastructure
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.