GSA / GSA/openacr

`validate` does not enforce the catalog's own "Not Evaluated … can only be used in WCAG Level AAA criteria" rule

Open
#366 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
JavaScript
Stars
128
Forks
36
PR merge metrics
No merged PRs in 30d

Description

### Environment

`@openacr/openacr@0.3.8` (npm), Node v24.13.0.

### Context

Every shipped catalog's `terms:` block defines Not Evaluated as: "The product has not been evaluated against the criterion. This can **only** be used in WCAG Level AAA criteria."

### Steps to reproduce

A document claiming `not-evaluated` on Level A criterion 1.1.1, validated **with** the catalog supplied:

```yaml
title: Not-Evaluated Level A Test
product:
name: Test Product
author:
email: test@example.com
catalog: 2.5-edition-wcag-2.2-508-en
chapters:
success_criteria_level_a:
criteria:
- num: "1.1.1"
components:
- name: web
adherence:
level: not-evaluated
notes: Not evaluated.
```

```console
$ openacr validate -f ne-level-a.yaml -c node_modules/@openacr/openacr/catalog/2.5-edition-wcag-2.2-508-en.yaml
Valid!
```

### Expected

`Invalid:` — the catalog's own term definition restricts `not-evaluated` to Level AAA criteria, and the validator has both the term and the chapter in hand.

### Actual

`Valid!`. The term-value check confirms `not-evaluated` is a catalog term but does not apply the AAA-only restriction, so a Level A/AA criterion can be silently left unevaluated in a document that passes the official validator.

### Why it matters

Combined with the absence of completeness checking (a document claiming only 2 of the ~50 A/AA criteria also validates), an ACR that dodges every hard requirement can pass the official toolchain end to end. Procurement reviewers reasonably read "validates" as stronger than it is.

Adjacent but distinct: #246 discussed the editor's None/Not-Evaluated default; this issue is about `validate` not enforcing the level restriction the catalogs themselves state.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the documented `openacr validate` command with the supplied catalog and Level A YAML. Trace the validator's term-value check and catalog chapter/criterion level handling; done means the same document is rejected as `Invalid` while an allowed Level AAA use remains valid.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript
Domain
accessibility, cli
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.