swiftlang / swiftlang/swift-testing

Provide an API for custom expectations with diagnostics

Open
#11 2 comments 4 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement public-api
Dominant language
Swift
Stars
2.2k
Forks
169
Avg merge
1d 18h
Merged PRs (30d)
56

Description

Description

Spawned from the discussion in https://forums.swift.org/t/brainstorming-customizing-matchers/67456/11:

When a custom comparison is needed, it was recommended that users could just write whatever comparison they need as a Bool-returning function and pass that into #expect. That's fine from the point of view of keeping the #expect API simple, but expectations will often want to provide better diagnostics than what can be extracted from the arguments.

For example, pretend we didn't have isSuperset(of:) and we wrote our own terrible predicate:

extension Collection where Element: Hashable {
  func isThisASuperset(of smaller: some Collection<Element>) -> Bool {
    let diff = Set(smaller).subtracting(Set(self))
    return diff.isEmpty
  }
}

@Test func foo() {
  let x = [1, 3]
  let y = [1, 2, 4, 5]
  #expect(x.isThisASuperset(of: y))
}

When running this test, the runner produces the following diagnostic:

Expectation failed: (x → [1, 3]).isItASuperset(of: y → [1, 2, 4, 5])

which is helpful! But what I'd really like is something like:

Expectation failed: argument contained [2, 4, 5] which were not in the receiver -- (x → [1, 3]).isItASuperset(of: y → [1, 2, 4, 5])

One way to achieve this would be to create an #expect overload that takes, say, an ExpectationResult instead. That could look like the following:

enum ExpectationResult {
  case success
  case failure(reason: String)
}

and then the predicate becomes:

extension Collection where Element: Hashable {
  func isThisASuperset(of smaller: some Collection<Element>) -> ExpectationResult {
    let diff = Set(smaller).subtracting(Set(self))
    return diff.isEmpty
      ? .success
      : .failure(reason: "argument contained \(diff) which were not in the receiver")
  }
}

This would be a bare minimum API for these kinds of diagnostics, and simple enough for most purposes. It wouldn't be advanced enough to factor in the named of the arguments though; if we wanted the output to be something like this, we'd need something more complex:

Expectation failed: 'y' contained [2, 4, 5] which were not in 'x' -- (x → [1, 3]).isItASuperset(of: y → [1, 2, 4, 5])

Expected behavior

No response

Actual behavior

No response

Steps to reproduce

No response

swift-testing version/commit hash

0.0.0-initial

Swift & OS version (output of swift --version && uname -a)

No response

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the linked Swift Forums discussion and the current #expect API. Compare the proposed ExpectationResult approach with the need to preserve existing expression diagnostics and support richer custom messages. Done means the project has a decided API design for custom expectations with diagnostics, together with coverage for successful and failing custom comparisons.

Written by the indexing model from the issue text.

Assessment

Tech stack
swift
Domain
testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.