swiftlang / swiftlang/github-workflows
Android testing is broken with swiftbuild
Nobody has claimed this yet.
- Dominant language
- Swift
- Stars
- 115
- Forks
- 57
- Avg merge
- 1d 8h
- Merged PRs (30d)
- 3
Description
In https://github.com/swiftlang/github-workflows/pull/269, the Android build script dropped the --build-system native flag, so we will use whatever the default is for the current toolchain (which, as of 6.4, is 'swiftbuild'). That PR rightly handles the fact that the native build created test executables with the .xctest suffix, but the new swiftbuild created the binaries ending with -test-runner (as discussed at https://github.com/swiftlang/swift-package-manager/issues/9673):
However, it doesn't take into account the fact that while native created a single monolithic test executable, swiftbuild appears to create a separate executable for each test target (@owenv confirm?)
For example, for the swift-numerics package:
native: .build/aarch64-unknown-linux-android28/debug/swift-numericsPackageTests.xctest
swiftbuild: .build/out/Products/Debug-android/ComplexTests-test-runner, .build/out/Products/Debug-android/RealTests-test-runner, .build/out/Products/Debug-android/IntegerUtilitiesTests-test-runner
This means that even if the test is being executed at all, it is likely only executing one of them. This may explain why swift-algorithms testing is passing with 6.3 Android, but failing with nightly-main Android.
Unless there is some hidden way to run all the tests as a single batch, I suspect that we'll need to have the action create a little script that executes each of the test executables, remembers the process result for each of them, and then passes/fails based on whether all of the individual tests passed. Something like:
# Run each of the test binaries, both in XCTest mode and Swift Testing mode
.build/out/Products/Debug-android/ComplexTests-test-runner; echo $? >> results.txt
.build/out/Products/Debug-android/ComplexTests-test-runner --testing-library xctest; echo $? >> results.txt
.build/out/Products/Debug-android/RealTests-test-runner; echo $? >> results.txt
.build/out/Products/Debug-android/RealTests-test-runner --testing-library xctest; echo $? >> results.txt
.build/out/Products/Debug-android/IntegerUtilitiesTests-test-runner; echo $? >> results.txt
.build/out/Products/Debug-android/IntegerUtilitiesTests-test-runner --testing-library xctest; echo $? >> results.txt
# grep for any non-0 non-69 values in results.txt and fail if any are found…
CC @swiftlang/android-workgroup as discussed at the workgroup meeting today
CC @dschaefer in case this is of interest for packaging and tooling (https://github.com/swiftlang/swift-package-manager/issues/9940#issuecomment-4361950153)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with .github/workflows/scripts/android/android-emulator-tests.sh at the linked lines and inspect how the Android test executable is selected and run. Compare the swiftbuild output under .build/out/Products/Debug-android with the native layout, including XCTest and Swift Testing modes. Done means every test-target runner executes and the workflow fails when any individual result is unsuccessful.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- android, github-actions, shell, swift
- Domain
- ci-cd, mobile, testing
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 58/100