Testing: "samples/" folder in swagger-codegen-generators

For the moment the only integration tests we have are located in the swagger-codegen repo. There are a lot of *.sh and *.bat scripts under bin/ that calls the cli tool and that generate code in samples/. Then each generated project is tested. This is great and this effort should continue.

At swagger-codegen-generators level we need something else between unit tests and the mentioned integration tests. We need to see the impact of a change in the Codegen (and in the abstract Codegen like DefaultCodegenConfig) in the generated code (for each supported framework).

I propose following approach:

We create a Unit Test class (say GenerateSampleTest) containing one test case (say generateXxxxTest()) for each kind of generator we want to support (by genarator I mean Codegen class & library combination. For example for JavaClientCodegen we need multiple test cases to support the different libraries). This calls the generator programatically (no dependency on cli tool) and generate code in a subfolder of the samples/ folder at the root of the swagger-codegen-generators repository (similar to the integration test suite in the swagger-codegen repo).

The idea is that after a change, the GenerateSampleTest Test will be called (I guess everybody runs a mvn verify locally). The idea is to control the changes in the samples/ folder and to commit them inside the same pull request.

The spec used as input should be in the openapi: 3.0.1 format. It should also be extended when new buggy pattern are detected to be as complete as possible. Over the last weeks, following issues were fixed for special cases:

• https://github.com/swagger-api/swagger-codegen/issues/7592
• https://github.com/swagger-api/swagger-codegen/issues/7754
• https://github.com/swagger-api/swagger-codegen/issues/7812
• https://github.com/swagger-api/swagger-codegen/issues/7853
• https://github.com/swagger-api/swagger-codegen/issues/7918
• https://github.com/swagger-api/swagger-codegen-generators/issues/9

The input spec used by the GenerateSampleTest should contains one example of each cases described in theses bug reports. Yes it will be a monster spec, but this is the point. For more realistic example, we have the integration test suite in swagger-codegen.

At CI (Travis) level, the idea is to break the build if after the execution of the Maven Build (that will run the GenerateSampleTest) the samples/ folder contains changes that are not committed. This way we ensure that the folder is updated with each Pull Request.

Before I start to implement it, I would like to know what you think.

cc: @HugoMario, @webron, @wing328

Review the proposed GenerateSampleTest, the existing samples/ layout, and the swagger-codegen integration scripts under bin/. Start by tracing the Maven verify flow and how Travis runs it. Done means generators produce committed samples/ updates from the OpenAPI 3.0.1 spec and CI detects uncommitted changes.