`validate` does not enforce the catalog's own "Not Evaluated … can only be used in WCAG Level AAA criteria" rule
- Dominant language
- JavaScript
- Stars
- 128
- Forks
- 36
- PR merge metrics
- No merged PRs in 30d
Description
### Environment
`@openacr/openacr@0.3.8` (npm), Node v24.13.0.
### Context
Every shipped catalog's `terms:` block defines Not Evaluated as: "The product has not been evaluated against the criterion. This can **only** be used in WCAG Level AAA criteria."
### Steps to reproduce
A document claiming `not-evaluated` on Level A criterion 1.1.1, validated **with** the catalog supplied:
```yaml
title: Not-Evaluated Level A Test
product:
name: Test Product
author:
email: test@example.com
catalog: 2.5-edition-wcag-2.2-508-en
chapters:
success_criteria_level_a:
criteria:
- num: "1.1.1"
components:
- name: web
adherence:
level: not-evaluated
notes: Not evaluated.
```
```console
$ openacr validate -f ne-level-a.yaml -c node_modules/@openacr/openacr/catalog/2.5-edition-wcag-2.2-508-en.yaml
Valid!
```
### Expected
`Invalid:` — the catalog's own term definition restricts `not-evaluated` to Level AAA criteria, and the validator has both the term and the chapter in hand.
### Actual
`Valid!`. The term-value check confirms `not-evaluated` is a catalog term but does not apply the AAA-only restriction, so a Level A/AA criterion can be silently left unevaluated in a document that passes the official validator.
### Why it matters
Combined with the absence of completeness checking (a document claiming only 2 of the ~50 A/AA criteria also validates), an ACR that dodges every hard requirement can pass the official toolchain end to end. Procurement reviewers reasonably read "validates" as stronger than it is.
Adjacent but distinct: #246 discussed the editor's None/Not-Evaluated default; this issue is about `validate` not enforcing the level restriction the catalogs themselves state.
Contributor guide
Research direction
Start by reproducing the documented `openacr validate` command with the supplied catalog and Level A YAML. Trace the validator's term-value check and catalog chapter/criterion level handling; done means the same document is rejected as `Invalid` while an allowed Level AAA use remains valid.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript
- Domain
- accessibility, cli
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100