swiftlang / swiftlang/github-workflows

Android testing is broken with swiftbuild

Open
#277 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Swift
Stars
115
Forks
57
Avg merge
1d 8h
Merged PRs (30d)
3

Description

In https://github.com/swiftlang/github-workflows/pull/269, the Android build script dropped the --build-system native flag, so we will use whatever the default is for the current toolchain (which, as of 6.4, is 'swiftbuild'). That PR rightly handles the fact that the native build created test executables with the .xctest suffix, but the new swiftbuild created the binaries ending with -test-runner (as discussed at https://github.com/swiftlang/swift-package-manager/issues/9673):

https://github.com/swiftlang/github-workflows/blob/0366eba41655706c5a306fbc0f798eb2ad6aa3e3/.github/workflows/scripts/android/android-emulator-tests.sh#L163-L175

However, it doesn't take into account the fact that while native created a single monolithic test executable, swiftbuild appears to create a separate executable for each test target (@owenv confirm?)

For example, for the swift-numerics package:

native: .build/aarch64-unknown-linux-android28/debug/swift-numericsPackageTests.xctest
swiftbuild: .build/out/Products/Debug-android/ComplexTests-test-runner, .build/out/Products/Debug-android/RealTests-test-runner, .build/out/Products/Debug-android/IntegerUtilitiesTests-test-runner

This means that even if the test is being executed at all, it is likely only executing one of them. This may explain why swift-algorithms testing is passing with 6.3 Android, but failing with nightly-main Android.

Unless there is some hidden way to run all the tests as a single batch, I suspect that we'll need to have the action create a little script that executes each of the test executables, remembers the process result for each of them, and then passes/fails based on whether all of the individual tests passed. Something like:

# Run each of the test binaries, both in XCTest mode and Swift Testing mode
.build/out/Products/Debug-android/ComplexTests-test-runner; echo $? >> results.txt
.build/out/Products/Debug-android/ComplexTests-test-runner --testing-library xctest; echo $? >> results.txt

.build/out/Products/Debug-android/RealTests-test-runner; echo $? >> results.txt
.build/out/Products/Debug-android/RealTests-test-runner --testing-library xctest; echo $? >> results.txt

.build/out/Products/Debug-android/IntegerUtilitiesTests-test-runner; echo $? >> results.txt
.build/out/Products/Debug-android/IntegerUtilitiesTests-test-runner --testing-library xctest; echo $? >> results.txt

# grep for any non-0 non-69 values in results.txt and fail if any are found…

CC @swiftlang/android-workgroup as discussed at the workgroup meeting today
CC @dschaefer in case this is of interest for packaging and tooling (https://github.com/swiftlang/swift-package-manager/issues/9940#issuecomment-4361950153)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with .github/workflows/scripts/android/android-emulator-tests.sh at the linked lines and inspect how the Android test executable is selected and run. Compare the swiftbuild output under .build/out/Products/Debug-android with the native layout, including XCTest and Swift Testing modes. Done means every test-target runner executes and the workflow fails when any individual result is unsuccessful.

Written by the indexing model from the issue text.

Assessment

Tech stack
android, github-actions, shell, swift
Domain
ci-cd, mobile, testing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.