NVIDIA / NVIDIA/cloudai

env_params: support full dot-path keys from TestDefinition (cmd_args.foo.bar, bench_cmd_args.*)

Open
#950 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
99
Forks
62
Avg merge
6d 12h
Merged PRs (30d)
17

Description

Summary

Today env_params keys must name a top-level cmd_args field. They cannot address nested attributes (cmd_args.foo.bar) or other CLI argument bags some workloads expose (e.g. vllm's bench_cmd_args). This makes the env_params path semantics inconsistent with the rest of the parsing surface.

Proposed by @podkidyshev in review of #901 (thread). Explicitly out of scope for #901; tracked here as a follow-up.

Current behavior

TestDefinition.validate_env_params rejects any key that is not a top-level cmd_args field:

unknown = sorted(k for k in self.env_params if k not in cmd_args_fields)
if unknown:
    raise ValueError(f"env_params keys {unknown} are not cmd_args fields on ...")

and is_env_sampled / is_dse_job match on the bare key, so nested and non-cmd_args paths are unreachable.

Proposed behavior

Let an env_params key be a full dot-path rooted at TestDefinition, so it can target any nested attribute consistently:

num_nodes = 2

[cmd_args]
  foo = 1

  [cmd_args.args]
  bar = [1, 2, 3]

[env_params]
"cmd_args.args.bar" = { weights = [0.1, 0.1, 0.8] }

This makes env_params paths consistent with how other features address nested config, and unblocks workloads whose sweepable knobs live outside the root cmd_args (e.g. bench_cmd_args).

Scope notes
  • Path resolution must be shared with is_env_sampled / is_dse_job / param_space / apply_params_set so all of them agree on what a key addresses.
  • Keep the existing leaf/list/weights validation (a key must resolve to a leaf candidate list).
  • Backward compatibility: bare top-level keys should keep working (or a clear migration).

cc @podkidyshev (offered to take this).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with TestDefinition.validate_env_params and trace how env_params keys are handled by is_env_sampled, is_dse_job, param_space, and apply_params_set. Define one shared full dot-path resolution flow that preserves bare top-level keys and existing leaf/list/weights validation. Done means nested TestDefinition attributes, including non-cmd_args argument bags such as bench_cmd_args, are addressed consistently by all named methods.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
tooling
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.