json-schema-org / json-schema-org/JSON-Schema-Test-Suite

Tests for format should make better use of vocabularies and optional/ is too overloaded with meaning

Open
#495 13 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

needs decision
Dominant language
Python
Stars
750
Forks
307
Avg merge
3d 17h
Merged PRs (30d)
24

Description

The draft2019-09 and draft2020-12 directories contain various tests for 'format'. The ones located in format.json document the results of everything (even bad formats) as valid: true. This is because of the movement of format to annotations by default. That's fine. So, these tests should all return valid for these schema versions.

However, the tests in the optional/format directory are setup with 'valid: false'. As if the results of the tests should be an assertion. But there isn't any data that determines this distinction other than their location. When evaluating the files in optional/format, the test suite should consider format failures as assertions. When evaluating the tests outside of optional/format, it should consider format failures as annotations.

Makes it a bit hard to auto generate test implementations given the test suite JSON files.

It would be nice if there was some distinction in the test suite. Perhaps actually including $vocabulary: { "...format": true } in the schema node within the tests. Since this should be the thing that governs the behavior.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Compare the test files in the draft2019-09 and draft2020-12 format.json directories with those in optional/format, focusing on how the schema node and valid values represent format failures. Determine how a vocabulary declaration such as $vocabulary in the schema node could distinguish assertion tests from annotation tests, then update the test-suite data and expected behavior so the distinction is explicit.

Written by the indexing model from the issue text.

Assessment

Domain
testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.