Non-Default Module Testing

We need to go beyond default module testing and dive changing property values.
Reason can be seen at https://github.com/checkstyle/checkstyle/issues/3656#issuecomment-267135176 .

We can already identify properties, but how are going to decide what values to set? Booleans are easy since they only have 2 values, but what about regular expression strings or files.
Should we hardcode some possible values and randomize the rest? Can we scan tests for property values?

How will testing multiple versions of the same module be handled for reporting as we need to know the results for the specific version of the module being checked.
If we build multiple configurations for multiple module versions, regression runtime will take longer.

We can request changes to make this faster/better in main checkstyle repo if it is reasonable.

This is blocked by default module testing.