Automattic / Automattic/Automattic-Tracks-iOS
Inconsistent macOS unit tests failures in CI
- Dominant language
- Swift
- Stars
- 47
- Forks
- 15
- Avg merge
- 7h 58m
- Merged PRs (30d)
- 1
Description
You may have noticed _some_ builds failing the macOS tests step. Emphasis on some, because this doesn't seem to be related to the code but to the agent running the builds.
### The failure
```
[16:45:44]: ▸ Build Succeeded
| [16:45:44]: ▸ ❌ error: Could not compute signing task inputs - operation was cancelled. (in target 'AutomatticTracksiOS_AutomatticTracksModelObjC' from project 'AutomatticTracksiOS')
| [16:45:44]: ▸ ❌ error: Could not compute signing task inputs - operation was cancelled. (in target 'AutomatticRemoteLogging' from project 'AutomatticTracksiOS')
| [16:45:44]: ▸ ❌ error: Could not compute signing task inputs - operation was cancelled. (in target 'AutomatticTracksConstantsObjC' from project 'AutomatticTracksiOS')
| [16:45:44]: ▸ ❌ error: Could not compute signing task inputs - operation was cancelled. (in target 'AutomatticTracksModelObjC' from project 'AutomatticTracksiOS')
| [16:45:44]: ▸ ❌ error: Could not compute signing task inputs - operation was cancelled. (in target 'AutomatticTracksiOS_AutomatticTracksTests' from project 'AutomatticTracksiOS')
| [16:45:44]: ▸ ❌ error: Could not compute signing task inputs - operation was cancelled. (in target 'AutomatticEncryptedLogs' from project 'AutomatticTracksiOS')
| [16:45:44]: ▸ ❌ error: Could not compute signing task inputs - operation was cancelled. (in target 'AutomatticTracksEventsForSwift' from project 'AutomatticTracksiOS')
| [16:45:44]: ▸ ❌ error: Could not compute signing task inputs - operation was cancelled. (in target 'AutomatticCrashLoggingUI' from project 'AutomatticTracksiOS')
| [16:45:44]: ▸ ❌ error: Could not compute signing task inputs - operation was cancelled. (in target 'AutomatticTracksModel' from project 'AutomatticTracksiOS')
| [16:45:44]: ▸ ❌ error: Could not compute signing task inputs - operation was cancelled. (in target 'AutomatticTracks' from project 'AutomatticTracksiOS')
| [16:45:44]: ▸ ❌ error: Could not compute signing task inputs - operation was cancelled. (in target 'AutomatticTracksEvents' from project 'AutomatticTracksiOS')
| [16:45:44]: ▸ ❌ error: Could not compute signing task inputs - operation was cancelled. (in target 'AutomatticTracksTestsObjC' from project 'AutomatticTracksiOS')
| [16:45:44]: ▸ ❌ error: Could not compute signing task inputs - operation was cancelled. (in target 'AutomatticTracksTests' from project 'AutomatticTracksiOS')
| [16:45:44]: ▸ ❌ error: Could not compute signing task inputs - operation was cancelled. (in target 'AutomatticExperiments' from project 'AutomatticTracksiOS')
| [16:45:44]: ▸ Testing failed:
| [16:45:44]: ▸ Could not compute signing task inputs - operation was cancelled.
| [16:45:44]: ▸ Testing cancelled because the build failed.
| [16:45:44]: ▸ ** TEST FAILED **
```
A quick Google search for this brings up a few results (e.g. [this](https://stackoverflow.com/questions/71776905/how-to-fix-error-could-not-compute-signing-task-inputs-operation-was-cancelle), [this](https://stackoverflow.com/questions/68478156/xcode-ui-tests-failing-to-build-in-ci-with-error-could-not-compute-signing-task), and [this](https://developer.apple.com/forums/thread/685492)) but all related to running tests via Fastlane on a Swift Package.
### The inconsistency
_It works on my machine..._ But not only that, the same commit can pass on one agent and fail on another. For example [`10f02d`](https://github.com/Automattic/Automattic-Tracks-iOS/commit/10f02d009dc69080d694836530dfc9e8dd4466dc)

and [`ef5cc5a`](https://github.com/Automattic/Automattic-Tracks-iOS/commit/ef5cc5a96234abdac7f9b2e280af2d9fbd3a5d86)

I wonder if this has something to do with the agents? Although I don't recall there being any partial update of our fleet. **Update:** This turned out not to be the case, see details below.
Details
To verify whether we're facing an agent-based issue, I thought we could list failed builds and their agents. If the certain agents consistently pass or fail, that would be an indicator to further investigate the theory.
Agent | Failed Build(s) | Passed Build(s)
--- | --- | ---
`001` | | https://buildkite.com/automattic/tracks-ios/builds/179
`002` | https://buildkite.com/automattic/tracks-ios/builds/179 | https://buildkite.com/automattic/tracks-ios/builds/177 (notice it was before all other failures, though, 2022/08/26 vs 08/29+ 🤔 )
`004` | https://buildkite.com/automattic/tracks-ios/builds/167 | https://buildkite.com/automattic/tracks-ios/builds/198
`007` | https://buildkite.com/automattic/tracks-ios/builds/180 |
`013` | https://buildkite.com/automattic/tracks-ios/builds/167 |
`014` | | https://buildkite.com/automattic/tracks-ios/builds/167, https://buildkite.com/automattic/tracks-ios/builds/191
~~Notice that the table is quite empty right now. This repository doesn't see much action.~~
**Update:** As you can see, the `MV-MKE-X64-004` agent has a failed and a successful build, so clearly the agent-based theory was wrong.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.