json-schema-org / json-schema-org/JSON-Schema-Test-Suite
Document the test inclusion guidance/criteria
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 750
- Forks
- 307
- Avg merge
- 3d 17h
- Merged PRs (30d)
- 24
Description
From https://github.com/json-schema-org/JSON-Schema-Test-Suite/issues/436#issuecomment-699685450 --
The bar is low for new additions to the suite. As broad guidelines:
- additional tests MUST be correct behavior as far as is dictated by the specification
- additional tests MUST NOT be impossible to run for an implementation that is correct under the specification
- additional tests which appear at the top-level of each version folder MUST test required behavior for all implementations. In particular they MUST NOT rely on behavior which the specification proscribes via
SHOULD,MAYRECOMMENDEDor equivalent. - additional tests MUST NOT attempt to clarify the specification itself independently for behavior that was not considered or proscribed by the specification. In the case of ambiguous text in the specification, the specification team SHOULD be consulted to confirm what behavior was intended. If the relevant scenario was clearly and specifically considered but the wording was unclear, tests MAY be added. Otherwise, the test MUST be deferred (i.e. not added with any expected result) until a specification with explicit decision on its behavior is published.
- additional tests SHOULD NOT be directly covered by already existing tests
- additional tests SHOULD be in "minimal form" -- by which we call a test "minimal" if it does not include additional behavior beyond which is required for the behavior the test covers, regardless of whether the schema is stylistically optimal
- additional tests SHOULD prefer "simpler" keywords to more complicated ones when testing unrelated behavior. As a concrete example, a test for the
anyOfkeyword will need to reference other keywords as part of its schema. It should prefer to use "simple" keywords likeconstwhenever possible, rather than more complex (to implement) ones likepatternPropertiesor$ref. This allows implementers access to more tests even whilst their implementation is incomplete or in-progress. - large numbers of autogenerated or programmatically generated tests which cover many relatively similar scenarios SHOULD NOT be added. Instead, a small representative sample should be added, with the full generative script made available for those who wish to run all combinations.
- additional tests MUST NOT break or modify API guarantees of the test suite itself without specific discussion and migration plan (this relies on #223, but the point here as an example is "you can't add tests in which the object in the schema property of the test is not a valid JSON Schema. This is a guarantee that all tests follow intentionally, and which was intended to be true indefinitely. If we want to add tests for scenarios in which that does not hold, it must be after discussion and design.")
- additional tests MUST NOT reference external schemas or resources other than within the
refRemote.jsonfile.
If for a particular set of tests the above is met, generally tests should be merged at which point:
- additional tests SHOULD NOT be merged so long as the sanity checks ("CI for the suite itself") is failing (which generally aims only to test some of the above API guarantees of the suite). If a bug in the sanity checks themselves is suspected, reviewers SHOULD file an issue to track the sanity check bug and MAY choose to merge the original PR immediately if they are confident that it is correct.
- additional tests MUST be added to all active drafts they apply to, including the
draft-next/folder, and authors SHOULD review each version to ensure the test is applicable before copying them.
Including some guidelines for modification or removal are likely also nice.
Proposed changes or improvements to the above are of course welcome via issue or discussion.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The issue names no documentation file; first locate the contributor or test-authoring guidance and review the existing sanity checks, active draft folders, and refRemote.json exception. Document the inclusion criteria, merge conditions, version-copying guidance, and proposed modification or removal guidance, then verify the result matches the suite's existing organization.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- json
- Domain
- documentation, testing-qa
- Issue type
- Documentation
- Difficulty
- 2/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100