google / google/xctestrunner

Add Support for Xcode 15 and iOS 17

Open
#60 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
158
Forks
69
PR merge metrics
No merged PRs in 30d

Description

We are trying to use xctestrunner to run testing on iOS 17 devices with xcode 15.0

We are running into the follow errors when passing a command:
First when running the following command:
`ios_test_runner --app_under_test_path ios-configurator.ipa --test_bundle_path ios-configurator-ui-tests.zip --work_dir ~/ios_test_runner_work_dir --test_type xcuitest test --id 00008020-001524340281002E`

we get the following error regardless of what iOS our device is on:
```
Command line invocation:
/Applications/Xcode15.0beta3.app/Contents/Developer/usr/bin/xcodebuild test-without-building -xctestrun ~/ios_test_runner_work_dir/TEST_ROOT/xctestrun.plist -destination id=00008020-001524340281002E -derivedDataPath /var/folders/6d/l0ggp9fj52g22q3x1ghl7bfh0000gq/T/tmp73pv379u -resultBundlePath /var/folders/6d/l0ggp9fj52g22q3x1ghl7bfh0000gq/T/tmp73pv379u/test.xcresult -collect-test-diagnostics=never

User defaults from command line:
collect-test-diagnostics = never
IDEBuildOperationResultBundlePath = /var/folders/6d/l0ggp9fj52g22q3x1ghl7bfh0000gq/T/tmp73pv379u/test.xcresult
IDEDerivedDataPathOverride = /var/folders/6d/l0ggp9fj52g22q3x1ghl7bfh0000gq/T/tmp73pv379u
IDEPackageSupportUseBuiltinSCM = YES

Writing result bundle at path:
/var/folders/6d/l0ggp9fj52g22q3x1ghl7bfh0000gq/T/tmp73pv379u/test.xcresult

xcodebuild: error: Failed to build workspace temporary with scheme Transient Testing.: File is of an unsupported type; must be either an xctestrun file or xctestproducts bundle.
```

I am able to fix this issue by manually changing the name of the `xctestrun.plist` file to `test.xctestrun` and rerunning the command:
`/Applications/Xcode15.0beta3.app/Contents/Developer/usr/bin/xcodebuild test-without-building -xctestrun ~/ios_test_runner_work_dir/TEST_ROOT/test.xctestrun -destination id=00008020-001524340281002E -derivedDataPath /var/folders/6d/l0ggp9fj52g22q3x1ghl7bfh0000gq/T/tmp73pv379u -resultBundlePath /var/folders/6d/l0ggp9fj52g22q3x1ghl7bfh0000gq/T/tmp73pv379u/test1.xcresult -collect-test-diagnostics=never`

After doing this we run into another error:
```
Command line invocation:
/Applications/Xcode15.0beta3.app/Contents/Developer/usr/bin/xcodebuild test-without-building -xctestrun ~/ios_test_runner_work_dir/TEST_ROOT/test.xctestrun -destination id=00008020-001524340281002E -derivedDataPath /var/folders/6d/l0ggp9fj52g22q3x1ghl7bfh0000gq/T/tmp73pv379u -resultBundlePath /var/folders/6d/l0ggp9fj52g22q3x1ghl7bfh0000gq/T/tmp73pv379u/test1.xcresult -collect-test-diagnostics=never

User defaults from command line:
collect-test-diagnostics = never
IDEBuildOperationResultBundlePath = /var/folders/6d/l0ggp9fj52g22q3x1ghl7bfh0000gq/T/tmpnp_tmxyg/test3.xcresult
IDEDerivedDataPathOverride = /var/folders/6d/l0ggp9fj52g22q3x1ghl7bfh0000gq/T/tmpnp_tmxyg
IDEPackageSupportUseBuiltinSCM = YES

Writing result bundle at path:
/var/folders/6d/l0ggp9fj52g22q3x1ghl7bfh0000gq/T/tmpnp_tmxyg/test3.xcresult

Testing started
2023-07-06 16:21:39.428 xcodebuild[67560:380128] [MT] DVTDevice: Error locating DeviceSupport directory using Optional("arm64e") or Optional("arm64e"): nilError
2023-07-06 16:21:46.156 xcodebuild[67560:380128] [MT] IDETestOperationsObserverDebug: 11.913 elapsed -- Testing started completed.
2023-07-06 16:21:46.156 xcodebuild[67560:380128] [MT] IDETestOperationsObserverDebug: 0.000 sec, +0.000 sec -- start
2023-07-06 16:21:46.156 xcodebuild[67560:380128] [MT] IDETestOperationsObserverDebug: 11.913 sec, +11.913 sec -- end

Test session results, code coverage, and logs:
/var/folders/6d/l0ggp9fj52g22q3x1ghl7bfh0000gq/T/tmpnp_tmxyg/test3.xcresult
```
We do not run into this second error with iOS 16, only with iOS 17.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.