swiftlang / swiftlang/swift-testing

API to reliably detect if a test is running under Swift Testing or XCTest

Open
#475 7 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement public-api
Dominant language
Swift
Stars
2.2k
Forks
169
Avg merge
1d 18h
Merged PRs (30d)
56

Description

Description

Like #474, this was also discussed in a lab.

This issue is related to the second half of https://github.com/Quick/Nimble/issues/1145: Determining whether to report errors to Swift Testing or XCTest.

Background about how Nimble determines the test runner

Nimble nominally supports working with other test runners, so that if you wanted, you could use non-XCTest test runners.

The way Nimble works is it dynamically determines the test runner environment the first time an assertion fails. At that point, Nimble checks whether the test runner is XCTest, and then reports the issue to XCTest using the "appropriate" API[^1]. Then, Nimble will cache that check.

The way Nimble checks if the test runner is XCTest is by checking if the XCTestCase class can be found. Which probably needs to be updated. For example, I would expect that check to still pass if Swift Testing is the test runner, just because I would also expect XCTest to be loaded in to the test bundle even if it's not actually used. Especially during this period when Swift Testing is just being introduced, and I can expect a lot of wonkiness going on. So, I'll probably definitely update that check to be more like "is there a currently running XCTestCase"?

The actual ask here

In the lab I had, it was mentioned that the only API for determining if a Swift Test is running is Test.current. However, as you mentioned, Test.current is a task local value, so if someone uses a detached Task inside of the test, then Test.current will incorrectly return nil (as far as the semantics of checking "am I running inside of a test?" go).

What Nimble needs here is a way to reliably and positively identify whether or not it should report the issue to Swift Testing.

Other Possible Approaches for reporting issues (Nimble Specific)

Also in https://github.com/Quick/Nimble/issues/1145, I wrote up that another way that Nimble could determine whether or not to report to Swift Testing or XCTest is to assume that if there is no current running XCTestCase, then it should fall back to Swift Testing. Which probably would work in the interim, but I'd certainly appreciate a way to positively verify that Nimble should report the failure to Swift Testing.

Another approach I wrote up is to try to report to both Swift Testing and XCTest. So, just blasting that a test failure happened (or an error was encountered, or a polling expectation timed out, etc.) to both of them, and hoping that only one of them records the failure. That's also unappealing to me because, again, I'd like to positively verify the right place to report the issue and only report it once. But it's certainly an approach.

Gosh, Rachel, that's a lot. Is there a TL;DR?

I need a way to reliably and positively identify if Nimble (and other testing assertion tools) should report issues to Swift Testing or XCTest.

[^1]: Which just inspired me to write up https://github.com/Quick/Nimble/issues/1146, because Nimble should be using XCTestCase.recordFailure.

Expected behavior

No response

Actual behavior

No response

Steps to reproduce

No response

swift-testing version/commit hash

No response

Swift & OS version (output of swift --version && uname -a)

No response

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the semantics of Swift Testing's Test.current, especially the detached Task case described in the issue, and compare them with the XCTest detection discussion. Define and validate a reliable positive detection approach that reports each failure to the correct test runner exactly once; the payload names no repository files or tests to run.

Written by the indexing model from the issue text.

Assessment

Tech stack
swift
Domain
testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.