Proposal: Create new repo for openapi tests, similar to json schema test suite

This issue is a proposal that a new repo be created that stores tests of openapi payloads, request, responses, and documents. These tests will help implementation verify that they have built features correctly.

There is existing prior work here that applies to json schema only in the json schema test suite: https://github.com/json-schema-org/JSON-Schema-Test-Suite

Need for this solution:

• there are few tests here: https://github.com/OAI/OpenAPI-Specification/tree/main/tests/v3.1 and they do not provide full coverage for openapi features, they do not describe what is acceptable and unacceptable for openapi-only schema formats, they may not be well known in the community, they do not have coverage tests for different versions of the openapi spec
• what is allowed for int32/int64/float/double is not explicitly defined in the spec or the format registry (for example is X.0 allowed in for int32?). Implementers need this info when building.
• I attempted to add int64 tests into the json schema test suite and was told that that repo is not the right place for that test, because the format is openapi-specific and is not listed in the json schema spec.

Benefits:

• tooling providers could have a verifiable way of proving that their implementation is conforming to the requirements
• long term this should reduce the number of what is the correct way to implement x feature questions on this repo because you can point them to the existing test
• including these tests in a separate repo will allow implmenters to include that repo as a subpackage and easily test against it. (I currently do this with the json schema test suite here)
• General json schema 2020-12 tests should not be included here because they already exist in the json schema test suite. Refer users to that suite also. This reduces the maintenance workload.

Proposal Short Term:

• create a new repo, Openapi Test Suite in the https://github.com/OAI org
• have one folder for each version of the openapi test suite
• move the existing tests into that
• perhaps organize the tests in the same directory structure as the openap spec?
• add tests in components/schemas/format.json which includes the int64 format tests in: https://github.com/json-schema-org/JSON-Schema-Test-Suite/pull/711

I would be happy to add additional test for int32/float/double/int formats
@karenetheridge mentioned that she could contribute tests from https://github.com/karenetheridge/OpenAPI-Modern/tree/master/t

Open Questions:

• How thorough should these tests be?
  • My proposal is to incrementally add coverage bit by bit starting with easiest tests, which may be component schema formats.
• Are you okay using the json schema test suite format for the schema test files?
• What openapi versions will have tests?
• Do these tests mean that the tooling MUST/MAY/verb implement these features? (How does json schema handle this Julian Berman?)

Start by reviewing the existing tests in OAI/OpenAPI-Specification/tests/v3.1 and the JSON Schema Test Suite, including the referenced int64 work. Define the repository structure, supported OpenAPI versions, test format, and conformance expectations before implementation. Done means the proposal’s open questions are resolved and a scoped plan exists for the new test repository.