env_params: support full dot-path keys from TestDefinition (cmd_args.foo.bar, bench_cmd_args.*)
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 99
- Forks
- 62
- Avg merge
- 6d 12h
- Merged PRs (30d)
- 17
Description
Summary
Today env_params keys must name a top-level cmd_args field. They cannot address nested attributes (cmd_args.foo.bar) or other CLI argument bags some workloads expose (e.g. vllm's bench_cmd_args). This makes the env_params path semantics inconsistent with the rest of the parsing surface.
Proposed by @podkidyshev in review of #901 (thread). Explicitly out of scope for #901; tracked here as a follow-up.
Current behavior
TestDefinition.validate_env_params rejects any key that is not a top-level cmd_args field:
unknown = sorted(k for k in self.env_params if k not in cmd_args_fields)
if unknown:
raise ValueError(f"env_params keys {unknown} are not cmd_args fields on ...")
and is_env_sampled / is_dse_job match on the bare key, so nested and non-cmd_args paths are unreachable.
Proposed behavior
Let an env_params key be a full dot-path rooted at TestDefinition, so it can target any nested attribute consistently:
num_nodes = 2
[cmd_args]
foo = 1
[cmd_args.args]
bar = [1, 2, 3]
[env_params]
"cmd_args.args.bar" = { weights = [0.1, 0.1, 0.8] }
This makes env_params paths consistent with how other features address nested config, and unblocks workloads whose sweepable knobs live outside the root cmd_args (e.g. bench_cmd_args).
Scope notes
- Path resolution must be shared with
is_env_sampled/is_dse_job/param_space/apply_params_setso all of them agree on what a key addresses. - Keep the existing leaf/list/weights validation (a key must resolve to a leaf candidate list).
- Backward compatibility: bare top-level keys should keep working (or a clear migration).
cc @podkidyshev (offered to take this).
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with TestDefinition.validate_env_params and trace how env_params keys are handled by is_env_sampled, is_dse_job, param_space, and apply_params_set. Define one shared full dot-path resolution flow that preserves bare top-level keys and existing leaf/list/weights validation. Done means nested TestDefinition attributes, including non-cmd_args argument bags such as bench_cmd_args, are addressed consistently by all named methods.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- tooling
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100