NVIDIA / NVIDIA/cloudai

single_sbatch routing: rework agent-driven (DSE/live-RL) handling and mixed-job guard ordering

Open
#937 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
99
Forks
62
Avg merge
6d 12h
Merged PRs (30d)
17

Description

Context

handle_dry_run_and_run (src/cloudai/cli/handlers.py) routes scenarios as:

agent_driven = [_is_dse_or_live_rl(tr) for tr in test_scenario.test_runs]
if args.single_sbatch or not any(agent_driven):  # in this mode cases are unrolled using grid search
    handle_non_dse_job(runner, args)
    return 0
if all(agent_driven):
    return handle_dse_job(runner, args)
logging.error("Mixing agent-driven (DSE / live-RL) and plain jobs is not allowed.")
return 1

The args.single_sbatch short-circuit is evaluated before the mixed-job guard. This ordering predates the live-RL work (it keyed on tr.is_dse_job originally) and is benign for DSE, because SingleSbatchRunner deliberately grid-unrolls DSE param spaces (unroll_dse / all_trs / handle_dse). But it has two rough edges:

  1. Mixed agent-driven + plain under --single-sbatch bypasses the "mixing not allowed" error and silently routes everything through handle_non_dse_job.
  2. SingleSbatchRunner has no live-RL path — it only branches on tr.is_dse_job. A live_rl_mode job under --single-sbatch would silently run once as a static job, because live-RL needs the in-process GymServer loop, not a batch grid.

A narrow stopgap guard was added for case (2) (error on live_rl_mode + single_sbatch) with a TODO pointing at this issue. This issue tracks the proper fix.

Proposed fix (dedicated PR)

  • Reject mixed agent-driven + plain before the single_sbatch short-circuit.
  • Make single_sbatch semantics explicit per agent type: DSE -> grid-unroll (current behavior), live-RL -> unsupported/error (or a live-RL-aware single-sbatch path if ever needed).
  • Add regression tests covering single_sbatch=True for DSE, live-RL, plain, and mixed scenarios.

Files

  • src/cloudai/cli/handlers.py (handle_dry_run_and_run, _is_dse_or_live_rl)
  • src/cloudai/systems/slurm/single_sbatch_runner.py (all_trs, handle_dse)
  • tests/test_handlers.py (_run_routing + routing tests)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in src/cloudai/cli/handlers.py at handle_dry_run_and_run and _is_dse_or_live_rl, then inspect all_trs and handle_dse in src/cloudai/systems/slurm/single_sbatch_runner.py. Use the _run_routing tests in tests/test_handlers.py as the entry point. Done means mixed jobs are rejected before single_sbatch routing, with regression coverage for DSE, live-RL, plain, and mixed scenarios.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli, testing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
74/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.