NVIDIA / NVIDIA/cloudai

Scenario HTML report never records pass/fail status

Open
#1,026 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
99
Forks
62
Avg merge
6d 12h
Merged PRs (30d)
17

Description

Scenario HTML report never records pass/fail status

Describe the Bug

StatusReporter.generate_scenario_report() (src/cloudai/reporter.py) writes
the persistent <scenario>.html artifact from ReportItem objects, but
ReportItem only carries name, description, logs_path, and nodes,
there is no status field:

@dataclass
class ReportItem:
    """Enhanced report item for Slurm systems with node information."""

    name: str
    description: str
    logs_path: Optional[str] = None
    nodes: Optional[str] = None

general-report.jinja2's "Results" column reflects exactly that, it only
ever renders a logs link or "no logs", never a pass/fail state:

{% if item.logs_path %}
    <td><a href="{{ item.logs_path }}">logs</a></td>
{% else %}
    <td>no logs</td>
{% endif %}

The real pass/fail computation already exists, StatusReporter.print_summary()
calls tr.test.was_run_successful(tr) per test run and builds a rich table
with PASSED/FAILED, but that table is only captured to the log stream
(console.capture()), never written to disk or threaded into ReportItem/
the HTML template. So the terminal output shows accurate status, but the
saved HTML report, the artifact someone would actually keep, share, or open
later, does not.

Steps to Reproduce

Version/context: local checkout of NVIDIA/cloudai main, commit d50dfc5b.

Run the bundled Sleep scenario for real (not dry-run):

cloudai run \
  --system-config conf/common/system/standalone_system.toml \
  --tests-dir conf/common/test \
  --test-scenario conf/common/test_scenario/sleep.toml

Terminal output correctly shows all test cases PASSED via the Scenario results table. Open the generated <results>/sleep-scenario.html in that
same run's output directory:

grep -i "passed\|failed" <results-dir>/sleep-scenario.html

No matches. The saved report has a "Results" column with only a logs link,
nothing indicating whether the test passed or failed.

Expected Behavior

The persisted <scenario>.html report should show the same PASSED/FAILED
status already computed for the terminal summary, not just a logs link. A
reviewer opening the saved report later, without having seen the original
terminal output, currently cannot tell whether any test case actually
passed.

Additional Context

Found while doing real (non-dry-run) validation of CloudAI through a small
wrapper project, CloudAI Autotune.
Happy to open a small PR: add a status/error_message field to
ReportItem, populate it in ReportItem.from_test_runs via the same
was_run_successful(tr) call print_summary already makes, and render it
in general-report.jinja2's Results column alongside the existing logs
link. Wanted to check the shape is right before writing it.

AI was used for context and guidance while investigating and drafting this
report.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in src/cloudai/reporter.py by comparing ReportItem.from_test_runs with StatusReporter.print_summary(), especially the existing was_run_successful(tr) calculation. Then inspect general-report.jinja2's Results column. Run the bundled Sleep scenario command and verify the generated sleep-scenario.html contains PASSED or FAILED for each test case alongside the logs link.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
tooling
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.