NVIDIA / NVIDIA/cloudai

Support relative-path test references and lazy-load only referenced test/hook tomls

Open
#985 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Improvement
Dominant language
Python
Stars
99
Forks
62
Avg merge
6d 12h
Merged PRs (30d)
17

Description

Problem

Parser.parse() in src/cloudai/parser.py globs and fully parses every .toml in --tests-dir and every hook toml, even when a scenario only references a few of them. Filtering to the tests actually used happens after everything is already loaded and validated.

This is hard to fix in isolation. A scenario references a test by its internal name field via test_name, not by file path. The only way to know which file has which name is to open and parse it. There is no name to file index, so lazy loading has nothing to work from without a change to how tests are referenced.

Proposal

Two small, sequential PRs.

PR 1: relative path test references. Add an optional path based reference on TestRunModel, alongside the existing test_name lookup. Fully backward compatible. Existing scenarios keep working unchanged. A [[Tests]] entry could point directly at a toml relative to the scenario file instead of relying on directory wide name resolution.

PR 2: lazy load referenced tomls. Depends on PR 1. When every [[Tests]] entry in a scenario uses a path reference, skip the eager glob and parse of the whole --tests-dir and hooks dir, and load only the files actually referenced. If a scenario mixes path and name references, fall back to today's eager behavior for the name based ones. Full mixed mode laziness can be a later follow up.

Open questions

What should a relative path resolve against. The scenario file's own directory seems most intuitive, flagging it rather than assuming.

Today, duplicate name values are rejected across the whole --tests-dir. Once loading is scoped to only the referenced files, does that check still need to hold directory wide, or just among the files actually loaded for a run.

Out of scope

Not touching the X | list[X] sweeping annotation issue or the wider parsing refactor. Those need their own scoping discussion.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with Parser.parse() in src/cloudai/parser.py and inspect the TestRunModel definition and existing scenario parsing. Establish the relative-path reference behavior first, including its resolution base, then verify backward compatibility before considering lazy loading. Done means referenced TOMLs load correctly, mixed name/path references retain eager behavior, and the relevant parser tests pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.