google / google/xctestrunner

No way to get xcresult from running logic tests

Open
#26 7 comments 8 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
158
Forks
69
PR merge metrics
No merged PRs in 30d

Description

xctestrunner currently assumes that any test without a test host (a logic test) must be run directly via the `xctest` agent, and not via `xcodebuild test-without-building`. This means there's no way to get an xctestresult bundle from logic tests, making it much harder to analyze what happens during the run of a log test.

Changing `xctest_session.py` to not check `if test_type != ios_constants.TestType.LOGIC_TEST`, and unconditionally run logic tests via `xcodebuild` seems to work just fine, and is "just" slower:

Direct:
`Stats over 10 runs: max = 1.0s, min = 0.9s, avg = 0.9s, dev = 0.0s`

Xcodebuild:
`Stats over 10 runs: max = 2.9s, min = 2.7s, avg = 2.7s, dev = 0.1s`

Xcodebuild, with the sleeps shortened:
`Stats over 10 runs: max = 2.5s, min = 2.2s, avg = 2.2s, dev = 0.1s`

Would an option to run logic tests via `xcodebuild` be acceptable?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.