ESMCI / ESMCI/cime

feat(flux): convert walltime to minutes/FSD for batch_system="flux"

Open
#5,017 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
174
Forks
225
Avg merge
1d 16h
Merged PRs (30d)
14

Description

### Summary
Flux's `-t/--time-limit` accepts **minutes or Flux Standard Duration (FSD, RFC 23)** — not `HH:MM:SS`. Tuolumne is currently the only E3SM machine running Flux, so this is handled as a **Flux-specific code path** in `env_batch.py`, leaving the generic `format_time()`/`walltime_format` mechanism untouched (used as-is by other Slurm/PBS-style machines).

### Reference: Flux's accepted format
Per `flux-submit(1)`:
> `-t, --time-limit=MINUTES|FSD`
> Set a time limit for the job in either minutes or Flux standard duration (RFC 23). FSD is a floating point number with a single character units suffix (`s`, `m`, `h`, or `d`). The default unit is minutes when no unit is specified.

### Root cause recap
`CIME/utils.py::format_time()` re-arranges `H`/`M`/`S` fields positionally; it does not sum across units. E.g. `walltime_format="%M"` on `01:10:00` yields `10`, not `70`. This isn't a bug in `format_time()` itself — it's simply the wrong tool for Flux's minutes/FSD requirement, and other machines rely on its current field-rearranging behavior, so it should not be changed generically.

### Proposed fix
In `CIME/XML/env_batch.py::get_job_overrides()` (~line 484), add a Flux-specific branch that converts the resolved walltime directly, rather than reusing the generic `format_time()`/`walltime_format` path for this scheduler:

```python
if batch_system == "flux":
seconds = convert_to_seconds(walltime)
# convert to minutes and/or FSD as appropriate
walltime =
else:
walltime_format = self.get_value("walltime_format")
if walltime_format:
... # existing format_time() path, unchanged
```

Implementation should support **both** Flux-accepted formats:
- **Minutes** (plain integer, e.g. `70`)
- **FSD** (floating point + unit suffix `s`/`m`/`h`/`d`, e.g. `70m`, `1.17h`)

Exact choice of which format to emit by default (and any rounding behavior for partial minutes) is an implementation detail to work out in the PR.

### Out of scope
- Changing the generic `format_time()` field-rearranging behavior used by other (non-flux) machines.
- Rounding/ceiling-division correctness improvements to the general `walltime_format` path.

### Validation
- Unit tests covering `batch_system="flux"` walltime conversion for both minutes and FSD output paths, e.g. `01:10:00` → `70` (minutes) and equivalent FSD form.
- Confirm non-flux batch systems are unaffected (existing `format_time()`/`walltime_format` behavior unchanged).
- End-to-end: submit a Tuolumne test case with `env_batch.xml` walltime `01:10:00`; confirm the generated `flux batch`/`flux submit -t` value is valid and the enforced job time limit matches.

### References
- [Project](https://flux-framework.org/)
- [GitHub](https://github.com/flux-framework)
- [Docs](https://flux-framework.readthedocs.io/en/latest/)

Contributor guide

Open the contributing guide

Research direction

Start in CIME/XML/env_batch.py at get_job_overrides(), then compare the existing generic format_time() path in CIME/utils.py. Add focused tests for Flux minutes and FSD conversion, verify non-Flux behavior remains unchanged, and use env_batch.xml with a 01:10:00 walltime to confirm the generated flux batch or flux submit -t value and enforced limit.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
tooling
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
62/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.