GSA / GSA/openacr

`validate` without `-c` skips all catalog checks — nonexistent criteria report Valid!

Open
#363 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
JavaScript
Stars
128
Forks
36
PR merge metrics
No merged PRs in 30d

Description

### Environment

`@openacr/openacr@0.3.8` (npm), Node v24.13.0.

### Steps to reproduce

`bogus-criterion.yaml` — a minimal document claiming a criterion that does not exist in any catalog:

```yaml
title: Bogus Test
product:
name: Bogus
author:
email: t@example.com
catalog: 2.5-edition-wcag-2.2-508-en
chapters:
success_criteria_level_a:
criteria:
- num: "9.9.9"
components:
- name: web
adherence:
level: supports
notes: test
```

```console
$ openacr validate -f bogus-criterion.yaml
Valid!

$ openacr validate -f bogus-criterion.yaml -c node_modules/@openacr/openacr/catalog/2.5-edition-wcag-2.2-508-en.yaml
Invalid: criteria '9.9.9' is not included in 'Table 1: Success Criteria, Level A'
```

### Expected

Either (a) `validate` resolves the catalog from the document's own `catalog:` field when `-c` is omitted, or (b) it refuses/warns loudly that no catalog checks were performed.

### Actual

Without `-c`, the document's `catalog:` field is not loaded and no catalog checks run at all — criterion membership, term values, and chapter placement are unchecked. The bare invocation reports `Valid!` for a document claiming a nonexistent criterion, which reads as a full validation pass.

### Why it matters

`validate -f file.yaml` is the natural first invocation, and its `Valid!` is indistinguishable from a real pass. Anyone gating an ACR pipeline on the bare form is running schema-shape checks only without knowing it.

Found while building automated ACR-drafting tooling against the pinned package; happy to provide more detail.

Contributor guide

Open the contributing guide

Research direction

Start at the `openacr validate` CLI entry point and trace how `-c` selects `node_modules/@openacr/openacr/catalog/2.5-edition-wcag-2.2-508-en.yaml`. Reproduce both commands with `bogus-criterion.yaml`, then verify that omitting `-c` either loads the document's `catalog:` value or clearly reports that catalog checks were skipped; the invalid criterion must not report `Valid!`.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, node.js
Domain
cli, testing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
70/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.