swiftlang / swiftlang/swift-testing
Add a type for complex input generation that performs exploratory testing
Nobody has claimed this yet.
- Dominant language
- Swift
- Stars
- 2.2k
- Forks
- 169
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 56
Description
Description
In order to support testing patterns where randomized input generation is used to explore a program's state space (such as fuzz, differential, property-based, or mutation testing) we need to add some affordance to developers to specify the following:
- The next value to pass into a
@Test - A way to receive the prior value and test result as a result of executing
@Test - A way to receive additional information, such as changes in code coverage, from the last
@Test - A way to stop input generation based on some heuristic or predicate.
Having these 4 items would be the essential set of primitives to really streamline the creation of tools like QuickCheck, libFuzzer, etc. with Swift Testing.
Perhaps this could take the form of a protocol, potentially named TestCaseInputGenerator, that has the following shape:
protocol TestCaseInputGenerator {
associatedtype AdditionalContext
func shouldStop<InputType>(given priorRun: (InputType, TestResult)?, additionalContext: AdditionalContext?) -> Bool
func nextValue<InputType>(given priorRun: (InputType, TestResult)?, additionalContext: AdditionalContext?) -> InputType
}
AdditionalContext can encapsulate data of interest to the test; previously tried values, their results, and other data like code coverage, crashes, etc.
I'm filing this issue rather quickly, in between tasks, so apologizes in advance for oversights on the type signatures and compilability of the proposed API. :)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the proposed TestCaseInputGenerator protocol and the four required primitives: next input, prior value and result, additional context, and stopping criteria. Define what the Swift Testing API should expose for exploratory generation, then validate the design against fuzz, differential, property-based, and mutation testing use cases.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- swift
- Domain
- testing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100