swiftlang / swiftlang/swift-testing

Add a type for complex input generation that performs exploratory testing

Open
#25 6 comments 5 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Swift
Stars
2.2k
Forks
169
Avg merge
1d 18h
Merged PRs (30d)
56

Description

Description

In order to support testing patterns where randomized input generation is used to explore a program's state space (such as fuzz, differential, property-based, or mutation testing) we need to add some affordance to developers to specify the following:

  1. The next value to pass into a @Test
  2. A way to receive the prior value and test result as a result of executing @Test
  3. A way to receive additional information, such as changes in code coverage, from the last @Test
  4. A way to stop input generation based on some heuristic or predicate.

Having these 4 items would be the essential set of primitives to really streamline the creation of tools like QuickCheck, libFuzzer, etc. with Swift Testing.

Perhaps this could take the form of a protocol, potentially named TestCaseInputGenerator, that has the following shape:

protocol TestCaseInputGenerator {
    associatedtype AdditionalContext
    func shouldStop<InputType>(given priorRun: (InputType, TestResult)?, additionalContext: AdditionalContext?) -> Bool
    func nextValue<InputType>(given priorRun: (InputType, TestResult)?, additionalContext: AdditionalContext?) -> InputType
}

AdditionalContext can encapsulate data of interest to the test; previously tried values, their results, and other data like code coverage, crashes, etc.

I'm filing this issue rather quickly, in between tasks, so apologizes in advance for oversights on the type signatures and compilability of the proposed API. :)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the proposed TestCaseInputGenerator protocol and the four required primitives: next input, prior value and result, additional context, and stopping criteria. Define what the Swift Testing API should expose for exploratory generation, then validate the design against fuzz, differential, property-based, and mutation testing use cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
swift
Domain
testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.