Reject orphan and unconfigured directories in the case manifest
- Dominant language
- Rust
- Stars
- 53
- Forks
- 16
- Avg merge
- 4h 22m
- Merged PRs (30d)
- 46
Description
## Problem
The manifest validator added by #457 verifies every configured suite, but an accidentally misspelled or abandoned directory below `tests/cases/` can remain outside the manifest and silently receive no CI coverage.
## Scope
- Walk suite directories below `tests/cases/` and compare them with the directories derived from `cases.toml`.
- Ignore documented non-suite files and nested multi-file case directories.
- Report every orphan path in one actionable validation error.
- Add temporary-directory unit tests for valid, orphaned, and misspelled layouts.
## Completion criteria
- `case_manifest.py validate` fails when a new suite directory is not represented by the matrix.
- Multi-file cases such as `shared/test88/` remain valid.
- The error names the exact paths and explains how to register or remove them.
Related: #457
Contributor guide
Research direction
Start in case_manifest.py with the existing validate command, then trace how cases.toml becomes the configured suite directories under tests/cases/. Run the current validation and inspect the existing unit-test setup before adding temporary-directory coverage for valid, orphaned, misspelled, and nested multi-file layouts. Done means validate reports all orphan paths with registration or removal guidance while accepting shared/test88/.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ci-cd, testing, tooling
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 76/100