google / google/xctestrunner

xctestrunner incorrectly handles maxRuntimeVersion (15.255.255 → 15.3)

Open
#72 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
158
Forks
69
PR merge metrics
No merged PRs in 30d

Description

### Summary

`xctestrunner` incorrectly parses the `maxRuntimeVersion` of **iPhone 6s Plus** (and similar devices) as `15.3`, which makes it impossible to create a simulator with iOS 15.5.

### Steps to Reproduce

Run following command on the valid bazel project:

```bash
bazel test --ios_simulator_device='iPhone 6s Plus' --ios_simulator_version=15.5
```

### Actual Behavior

The test runner fails with:

```
Traceback (most recent call last):
File "/path/to/bazel//a59b5fbd1ac4fd988fa8e6c30761d359/sandbox/darwin-sandbox/487/execroot/_main/bazel-out/darwin_arm64-fastbuild/bin/Tests/Foundation/NSData_zlibTests/NSData_zlibUnitTest.runfiles/_main/../rules_apple++non_module_deps+xctestrunner/xctestrunner/test_runner/ios_test_runner.py", line 319, in
sys.exit(main(sys.argv))
^^^^^^^^^^^^^^
File "/path/to/bazel//a59b5fbd1ac4fd988fa8e6c30761d359/sandbox/darwin-sandbox/487/execroot/_main/bazel-out/darwin_arm64-fastbuild/bin/Tests/Foundation/NSData_zlibTests/NSData_zlibUnitTest.runfiles/_main/../rules_apple++non_module_deps+xctestrunner/xctestrunner/test_runner/ios_test_runner.py", line 313, in main
exit_code = args.func(args)
^^^^^^^^^^^^^^^
File "/path/to/bazel//a59b5fbd1ac4fd988fa8e6c30761d359/sandbox/darwin-sandbox/487/execroot/_main/bazel-out/darwin_arm64-fastbuild/bin/Tests/Foundation/NSData_zlibTests/NSData_zlibUnitTest.runfiles/_main/../rules_apple++non_module_deps+xctestrunner/xctestrunner/test_runner/ios_test_runner.py", line 216, in _SimulatorTest
return _RunSimulatorTest(args)
^^^^^^^^^^^^^^^^^^^^^^^
File "/path/to/bazel//a59b5fbd1ac4fd988fa8e6c30761d359/sandbox/darwin-sandbox/487/execroot/_main/bazel-out/darwin_arm64-fastbuild/bin/Tests/Foundation/NSData_zlibTests/NSData_zlibUnitTest.runfiles/_main/../rules_apple++non_module_deps+xctestrunner/xctestrunner/test_runner/ios_test_runner.py", line 186, in _RunSimulatorTest
simulator_id, _, os_version, _ = simulator_util.CreateNewSimulator(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/path/to/bazel//a59b5fbd1ac4fd988fa8e6c30761d359/sandbox/darwin-sandbox/487/execroot/_main/bazel-out/darwin_arm64-fastbuild/bin/Tests/Foundation/NSData_zlibTests/NSData_zlibUnitTest.runfiles/rules_apple++non_module_deps+xctestrunner/xctestrunner/simulator_control/simulator_util.py", line 358, in CreateNewSimulator
_ValidateSimulatorTypeWithOsVersion(device_type, os_version)
File "/path/to/bazel//a59b5fbd1ac4fd988fa8e6c30761d359/sandbox/darwin-sandbox/487/execroot/_main/bazel-out/darwin_arm64-fastbuild/bin/Tests/Foundation/NSData_zlibTests/NSData_zlibUnitTest.runfiles/rules_apple++non_module_deps+xctestrunner/xctestrunner/simulator_control/simulator_util.py", line 654, in _ValidateSimulatorTypeWithOsVersion
raise ios_errors.IllegalArgumentError(
xctestrunner.shared.ios_errors.IllegalArgumentError: The max OS version of iPhone 6s Plus is 15.300000. But current OS version is 15.5
error: tests exited with '1
```

### Root Cause

This happens because [xctestrunner/simulator_control/simtype_profile.py](https://github.com/google/xctestrunner/blob/8710f141dfb0a3efe2744f865f914783259c24b0/xctestrunner/simulator_control/simtype_profile.py#L100) does not handle **special `maxRuntimeVersion` patterns** correctly. On my machine:

```bash
$ xcrun simctl list devicetypes -j | python3 -c "
import json, sys
data = json.load(sys.stdin)
iphone_devices = [d for d in data['devicetypes'] if d['name'].startswith('iPhone')]

print('iPhone Runtime Version Analysis:')
print('=' * 50)
for device in iphone_devices:
max_runtime = device.get('maxRuntimeVersionString', 'unlimited')
print(f'{device[\"name\"]:30} Max Runtime: {max_runtime}')
"
```

shows:

```
iPhone Runtime Version Analysis:
==================================================
iPhone 16 Pro Max Runtime: 65535.255.255
iPhone 16 Pro Max Max Runtime: 65535.255.255
iPhone 16e Max Runtime: 65535.255.255
iPhone 16 Max Runtime: 65535.255.255
iPhone 16 Plus Max Runtime: 65535.255.255
iPhone 15 Pro Max Runtime: 65535.255.255
iPhone 15 Pro Max Max Runtime: 65535.255.255
iPhone 15 Max Runtime: 65535.255.255
iPhone 15 Plus Max Runtime: 65535.255.255
iPhone 14 Pro Max Runtime: 65535.255.255
iPhone 14 Pro Max Max Runtime: 65535.255.255
iPhone 14 Max Runtime: 65535.255.255
iPhone 14 Plus Max Runtime: 65535.255.255
iPhone SE (3rd generation) Max Runtime: 65535.255.255
iPhone 13 Pro Max Runtime: 65535.255.255
iPhone 13 Pro Max Max Runtime: 65535.255.255
iPhone 13 Max Runtime: 65535.255.255
iPhone 13 mini Max Runtime: 65535.255.255
iPhone 12 Pro Max Runtime: 65535.255.255
iPhone 12 Pro Max Max Runtime: 65535.255.255
iPhone 12 Max Runtime: 65535.255.255
iPhone 12 mini Max Runtime: 65535.255.255
iPhone SE (2nd generation) Max Runtime: 65535.255.255
iPhone 11 Pro Max Runtime: 65535.255.255
iPhone 11 Pro Max Max Runtime: 65535.255.255
iPhone 11 Max Runtime: 65535.255.255
iPhone Xʀ Max Runtime: 18.99.0
iPhone Xs Max Runtime: 18.99.0
iPhone Xs Max Max Runtime: 18.99.0
iPhone X Max Runtime: 16.9.0
iPhone 8 Max Runtime: 16.9.0
iPhone 8 Plus Max Runtime: 16.9.0
iPhone 7 Max Runtime: 15.255.255
iPhone 7 Plus Max Runtime: 15.255.255
iPhone SE (1st generation) Max Runtime: 15.255.255
iPhone 6s Max Runtime: 15.255.255
iPhone 6s Plus Max Runtime: 15.255.255
```

Considering the model's last supported iOS versions, the following patterns are used:

* `65535.255.255` → effectively unlimited
* `18.99.0` → upper bound pattern
* `15.255.255` → upper bound pattern

However, `xctestrunner` truncates these values incorrectly, causing `15.255.255` to be parsed as `15.3`.

### Expected Behavior

* `iPhone 6s Plus` should be treated as supporting up to **iOS 15.x**, so requesting `15.5` should succeed.
* Special version encodings like `255.255`, `99.0`, or `65535.255.255` should be parsed as **upper limits**, not literal decimal values.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.