NVIDIA / NVIDIA/simready-foundation

[ci] No tests run in CI; conformance validators and spec data are unverified

Open
#10 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
88
Forks
18
PR merge metrics
No merged PRs in 30d

Description

Summary

CI never runs the test suite, and there is no automated check that the spec data (feature manifests, profiles) and the conformance validators stay consistent. Correctness regressions in validators or dangling spec references can land silently.

Evidence

CI only builds docs. The single workflow .github/workflows/sphinx.yml builds and deploys Sphinx; no workflow runs pytest (grep for pytest across .github/workflows/ returns nothing). The repo has ~23 pytest files (under nv_core/package_sample/tests/ and nv_core/cip_specs/asset_handler_modules/.../transformer/tests/), but they are never executed in CI, and they cover the packaging/transformer workflow — not the per-capability conformance validators.

Conformance validators are untested. There are 23 capabilities/*/validation.py files with ~92 register_rule( / ~95 register_requirements( registrations. No unit tests exercise these rule classes (no test_* imports the capability validation modules). These validators are the enforcement of the spec — a silent break mislabels customer assets as conformant or not.

No spec-integrity check. Nothing verifies that:

  • requirement IDs listed in features/FET_*.json actually exist as requirement docs
  • features referenced in profiles.toml exist as manifests
  • requirement/feature IDs are well-formed
    A renamed or deleted requirement leaves dangling references that surface only by manual discovery.

Why it matters

The validators and the spec graph are the product. Without CI running tests or integrity checks, a green build guarantees only that the docs compiled — not that the conformance logic is correct or the spec references resolve.

Suggested resolution

Add a CI job (on PR/push) that:

  1. Runs the existing pytest suite.
  2. Adds unit tests for the conformance validators (at least one passing + one failing fixture per rule).
  3. Runs a spec-integrity linter (feature→requirement and profile→feature references resolve; ID format checks).
  4. Builds the docs with warnings-as-errors (-W) so undefined roles/directives fail the build — see the companion docs-rendering issue.

Verified on main @ 2026.04.1, 2026-05-28.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with .github/workflows/sphinx.yml and the existing pytest files under nv_core/package_sample/tests/ and nv_core/cip_specs/asset_handler_modules/.../transformer/tests/. Then inspect the 23 capabilities//validation.py files, features/FET_.json, and profiles.toml to define the validator and spec-integrity coverage. Done means CI runs pytest, validator fixtures and reference/ID checks pass, and Sphinx builds with warnings treated as errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, python
Domain
ci-cd, testing, tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.