json-schema-org / json-schema-org/JSON-Schema-Test-Suite
Tests for format should make better use of vocabularies and optional/ is too overloaded with meaning
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 750
- Forks
- 307
- Avg merge
- 3d 17h
- Merged PRs (30d)
- 24
Description
The draft2019-09 and draft2020-12 directories contain various tests for 'format'. The ones located in format.json document the results of everything (even bad formats) as valid: true. This is because of the movement of format to annotations by default. That's fine. So, these tests should all return valid for these schema versions.
However, the tests in the optional/format directory are setup with 'valid: false'. As if the results of the tests should be an assertion. But there isn't any data that determines this distinction other than their location. When evaluating the files in optional/format, the test suite should consider format failures as assertions. When evaluating the tests outside of optional/format, it should consider format failures as annotations.
Makes it a bit hard to auto generate test implementations given the test suite JSON files.
It would be nice if there was some distinction in the test suite. Perhaps actually including $vocabulary: { "...format": true } in the schema node within the tests. Since this should be the thing that governs the behavior.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Compare the test files in the draft2019-09 and draft2020-12 format.json directories with those in optional/format, focusing on how the schema node and valid values represent format failures. Determine how a vocabulary declaration such as $vocabulary in the schema node could distinguish assertion tests from annotation tests, then update the test-suite data and expected behavior so the distinction is explicit.
Written by the indexing model from the issue text.
Assessment
- Domain
- testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100