NVIDIA / NVIDIA/simready-foundation
[ci] No tests run in CI; conformance validators and spec data are unverified
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 88
- Forks
- 18
- PR merge metrics
- No merged PRs in 30d
Description
Summary
CI never runs the test suite, and there is no automated check that the spec data (feature manifests, profiles) and the conformance validators stay consistent. Correctness regressions in validators or dangling spec references can land silently.
Evidence
CI only builds docs. The single workflow .github/workflows/sphinx.yml builds and deploys Sphinx; no workflow runs pytest (grep for pytest across .github/workflows/ returns nothing). The repo has ~23 pytest files (under nv_core/package_sample/tests/ and nv_core/cip_specs/asset_handler_modules/.../transformer/tests/), but they are never executed in CI, and they cover the packaging/transformer workflow — not the per-capability conformance validators.
Conformance validators are untested. There are 23 capabilities/*/validation.py files with ~92 register_rule( / ~95 register_requirements( registrations. No unit tests exercise these rule classes (no test_* imports the capability validation modules). These validators are the enforcement of the spec — a silent break mislabels customer assets as conformant or not.
No spec-integrity check. Nothing verifies that:
- requirement IDs listed in
features/FET_*.jsonactually exist as requirement docs - features referenced in
profiles.tomlexist as manifests - requirement/feature IDs are well-formed
A renamed or deleted requirement leaves dangling references that surface only by manual discovery.
Why it matters
The validators and the spec graph are the product. Without CI running tests or integrity checks, a green build guarantees only that the docs compiled — not that the conformance logic is correct or the spec references resolve.
Suggested resolution
Add a CI job (on PR/push) that:
- Runs the existing pytest suite.
- Adds unit tests for the conformance validators (at least one passing + one failing fixture per rule).
- Runs a spec-integrity linter (feature→requirement and profile→feature references resolve; ID format checks).
- Builds the docs with warnings-as-errors (
-W) so undefined roles/directives fail the build — see the companion docs-rendering issue.
Verified on main @ 2026.04.1, 2026-05-28.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with .github/workflows/sphinx.yml and the existing pytest files under nv_core/package_sample/tests/ and nv_core/cip_specs/asset_handler_modules/.../transformer/tests/. Then inspect the 23 capabilities//validation.py files, features/FET_.json, and profiles.toml to define the validator and spec-integrity coverage. Done means CI runs pytest, validator fixtures and reference/ID checks pass, and Sphinx builds with warnings treated as errors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github-actions, python
- Domain
- ci-cd, testing, tooling
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100