ESMCI / ESMCI/cime

feat(flux): support omit_in_job attribute on submit_args to drop args invalid when submitting from inside a job

Open
#5,018 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
174
Forks
225
Avg merge
1d 16h
Merged PRs (30d)
14

Description

### Summary
On Tuolumne, `-p ` is required for the **initial** `flux batch`/`flux submit` call (issued from outside any Flux instance), but must be **dropped** when a submission is issued **from inside an already-running job** (a true in-job resubmit), since Flux gives each job its own ephemeral nested cluster instance with no partitions defined. Resubmitting with a stale `-p ` from within a running job fails because that partition doesn't exist in the new instance.

### Current CIME behavior
- `CIME/XML/env_batch.py::get_submit_args()` / `_resolve_argument()` (~lines 621-770) resolves `` from `config_batch.xml` identically regardless of submission context.
- The resubmit path (`submit_jobs`, `resubmit_immediate`/`submit_resubmits`, ~lines 855-998) doesn't currently distinguish "submitted from outside a job" vs. "submitted from within a job's own resubmit step."

### Proposed design
1. **New `` attribute: `omit_in_job`** (boolean, default `false`) — add to the `` schema in `config_batch.xsd`, settable on individual submit args in `config_batch.xml`. When `omit_in_job="true"`, the arg is dropped from the resolved submit command if the submission is happening from inside an already-running batch job.
2. **In-job detection via `batch_env`:** add support for checking common, well-known batch-system environment variables that indicate an active job (e.g. `SLURM_JOB_ID`, `PBS_JOBID`, `FLUX_JOB_ID`) as part of/alongside the existing `batch_env` handling in `env_batch.py`. Since batch schedulers change infrequently, this scheduler→env-var mapping can be a small **static/hardcoded list in CIME** rather than a fully config-driven mapping — no need for per-machine config to declare which env var to check.
3. In `get_submit_args()`/`_resolve_argument()`, use this in-job check to decide whether to skip any arg marked `omit_in_job="true"`.
4. Apply `omit_in_job="true"` to the `-p` arg in Tuolumne's flux `config_batch.xml` (model-config change, once schema/mechanism support lands in CIME).

### Scope of affected args
`-p` is the only confirmed case so far. Other submit args scoped to the outer/initial allocation may also need `omit_in_job="true"`, but this hasn't been tested yet on Tuolumne — the mechanism should be generic/attribute-driven (not hardcoded to `-p`) so additional args can opt in later without further code changes.

### Validation
- Unit test in `CIME/tests/test_unit_xml_env_batch.py`: resolve submit args in both "in-job" (env var present, e.g. `FLUX_JOB_ID` set) and "not-in-job" (env var absent) contexts against a flux-like config with an `omit_in_job="true"` arg; assert it's included only in the not-in-job case.
- End-to-end on Tuolumne with `RESUBMIT>=1`: confirm the initial `flux batch` includes `-p`, and the in-job resubmit call omits it.
- **Follow-up investigation (fast-follow, not blocking initial fix):** test on Tuolumne to identify whether other submit args besides `-p` need `omit_in_job="true"`.

### Open questions
- Confirm the static list of env vars to check per batch system (e.g. `SLURM_JOB_ID`, `PBS_JOBID`, `LSB_JOBID`, `FLUX_JOB_ID`) covers all currently supported CIME batch systems.
- Confirm exact integration point within `batch_env` handling vs. a small standalone helper reusing similar logic.

### References
- [Project](https://flux-framework.org/)
- [GitHub](https://github.com/flux-framework)
- [Docs](https://flux-framework.readthedocs.io/en/latest/)

Contributor guide

Open the contributing guide

Research direction

Start with CIME/XML/env_batch.py, especially get_submit_args(), _resolve_argument(), and the resubmit paths around submit_jobs and resubmit_immediate/submit_resubmits. Review config_batch.xsd and the flux config_batch.xml, then run the unit tests in CIME/tests/test_unit_xml_env_batch.py for both in-job and non-job environments; done means the marked argument is resolved correctly in both cases and the existing validation passes.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
devops
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.