swiftlang / swiftlang/swift-testing
Provide an API for custom expectations with diagnostics
Nobody has claimed this yet.
- Dominant language
- Swift
- Stars
- 2.2k
- Forks
- 169
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 56
Description
Description
Spawned from the discussion in https://forums.swift.org/t/brainstorming-customizing-matchers/67456/11:
When a custom comparison is needed, it was recommended that users could just write whatever comparison they need as a Bool-returning function and pass that into #expect. That's fine from the point of view of keeping the #expect API simple, but expectations will often want to provide better diagnostics than what can be extracted from the arguments.
For example, pretend we didn't have isSuperset(of:) and we wrote our own terrible predicate:
extension Collection where Element: Hashable {
func isThisASuperset(of smaller: some Collection<Element>) -> Bool {
let diff = Set(smaller).subtracting(Set(self))
return diff.isEmpty
}
}
@Test func foo() {
let x = [1, 3]
let y = [1, 2, 4, 5]
#expect(x.isThisASuperset(of: y))
}
When running this test, the runner produces the following diagnostic:
Expectation failed: (x → [1, 3]).isItASuperset(of: y → [1, 2, 4, 5])
which is helpful! But what I'd really like is something like:
Expectation failed: argument contained [2, 4, 5] which were not in the receiver -- (x → [1, 3]).isItASuperset(of: y → [1, 2, 4, 5])
One way to achieve this would be to create an #expect overload that takes, say, an ExpectationResult instead. That could look like the following:
enum ExpectationResult {
case success
case failure(reason: String)
}
and then the predicate becomes:
extension Collection where Element: Hashable {
func isThisASuperset(of smaller: some Collection<Element>) -> ExpectationResult {
let diff = Set(smaller).subtracting(Set(self))
return diff.isEmpty
? .success
: .failure(reason: "argument contained \(diff) which were not in the receiver")
}
}
This would be a bare minimum API for these kinds of diagnostics, and simple enough for most purposes. It wouldn't be advanced enough to factor in the named of the arguments though; if we wanted the output to be something like this, we'd need something more complex:
Expectation failed: 'y' contained [2, 4, 5] which were not in 'x' -- (x → [1, 3]).isItASuperset(of: y → [1, 2, 4, 5])
Expected behavior
No response
Actual behavior
No response
Steps to reproduce
No response
swift-testing version/commit hash
0.0.0-initial
Swift & OS version (output of swift --version && uname -a)
No response
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the linked Swift Forums discussion and the current #expect API. Compare the proposed ExpectationResult approach with the need to preserve existing expression diagnostics and support richer custom messages. Done means the project has a decided API design for custom expectations with diagnostics, together with coverage for successful and failing custom comparisons.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- swift
- Domain
- testing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100