feat(flux): support omit_in_job attribute on submit_args to drop args invalid when submitting from inside a job
- Dominant language
- Python
- Stars
- 174
- Forks
- 225
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 14
Description
### Summary
On Tuolumne, `-p ` is required for the **initial** `flux batch`/`flux submit` call (issued from outside any Flux instance), but must be **dropped** when a submission is issued **from inside an already-running job** (a true in-job resubmit), since Flux gives each job its own ephemeral nested cluster instance with no partitions defined. Resubmitting with a stale `-p ` from within a running job fails because that partition doesn't exist in the new instance.
### Current CIME behavior
- `CIME/XML/env_batch.py::get_submit_args()` / `_resolve_argument()` (~lines 621-770) resolves `` from `config_batch.xml` identically regardless of submission context.
- The resubmit path (`submit_jobs`, `resubmit_immediate`/`submit_resubmits`, ~lines 855-998) doesn't currently distinguish "submitted from outside a job" vs. "submitted from within a job's own resubmit step."
### Proposed design
1. **New `` attribute: `omit_in_job`** (boolean, default `false`) — add to the `` schema in `config_batch.xsd`, settable on individual submit args in `config_batch.xml`. When `omit_in_job="true"`, the arg is dropped from the resolved submit command if the submission is happening from inside an already-running batch job.
2. **In-job detection via `batch_env`:** add support for checking common, well-known batch-system environment variables that indicate an active job (e.g. `SLURM_JOB_ID`, `PBS_JOBID`, `FLUX_JOB_ID`) as part of/alongside the existing `batch_env` handling in `env_batch.py`. Since batch schedulers change infrequently, this scheduler→env-var mapping can be a small **static/hardcoded list in CIME** rather than a fully config-driven mapping — no need for per-machine config to declare which env var to check.
3. In `get_submit_args()`/`_resolve_argument()`, use this in-job check to decide whether to skip any arg marked `omit_in_job="true"`.
4. Apply `omit_in_job="true"` to the `-p` arg in Tuolumne's flux `config_batch.xml` (model-config change, once schema/mechanism support lands in CIME).
### Scope of affected args
`-p` is the only confirmed case so far. Other submit args scoped to the outer/initial allocation may also need `omit_in_job="true"`, but this hasn't been tested yet on Tuolumne — the mechanism should be generic/attribute-driven (not hardcoded to `-p`) so additional args can opt in later without further code changes.
### Validation
- Unit test in `CIME/tests/test_unit_xml_env_batch.py`: resolve submit args in both "in-job" (env var present, e.g. `FLUX_JOB_ID` set) and "not-in-job" (env var absent) contexts against a flux-like config with an `omit_in_job="true"` arg; assert it's included only in the not-in-job case.
- End-to-end on Tuolumne with `RESUBMIT>=1`: confirm the initial `flux batch` includes `-p`, and the in-job resubmit call omits it.
- **Follow-up investigation (fast-follow, not blocking initial fix):** test on Tuolumne to identify whether other submit args besides `-p` need `omit_in_job="true"`.
### Open questions
- Confirm the static list of env vars to check per batch system (e.g. `SLURM_JOB_ID`, `PBS_JOBID`, `LSB_JOBID`, `FLUX_JOB_ID`) covers all currently supported CIME batch systems.
- Confirm exact integration point within `batch_env` handling vs. a small standalone helper reusing similar logic.
### References
- [Project](https://flux-framework.org/)
- [GitHub](https://github.com/flux-framework)
- [Docs](https://flux-framework.readthedocs.io/en/latest/)
Contributor guide
Research direction
Start with CIME/XML/env_batch.py, especially get_submit_args(), _resolve_argument(), and the resubmit paths around submit_jobs and resubmit_immediate/submit_resubmits. Review config_batch.xsd and the flux config_batch.xml, then run the unit tests in CIME/tests/test_unit_xml_env_batch.py for both in-job and non-job environments; done means the marked argument is resolved correctly in both cases and the existing validation passes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- devops
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100