CI: MAUI R2R Helix device-lab failures across Android queues
- Dominant language
- C#
- Stars
- 2.1k
- Forks
- 579
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 257
Description
### Android framework version
.NET 11 / `net11.0-android` CI builds.
### Affected platform version
Helix queues:
- `windows.11.amd64.android.open` (Android ARM64 devices)
- `windows.11.amd64.android-armel.open` (Android ARM32 devices)
Sample window: the latest 100 completed public `dotnet-android` pipeline builds at the time of investigation, from 2026-08-31 through 2026-09-03.
### Description
Recent non-fork PR builds are frequently failing in `Run MAUI R2R APKs on Helix`. These failures are not Helix submission/authentication failures and do not demonstrate MAUI or ReadyToRun regressions: each job was submitted and scheduled successfully, then failed on device setup before the APK launch and `pidof` smoke check.
Of 92 sampled builds containing the task, 72 passed, 19 failed, and 1 was skipped. Eight of the failures were the separately understood fork-token problem. The remaining 11 direct-PR builds contained 12 failed work items:
- 8 Android package-manager/UID exhaustion failures
- 3 ADB authorization timeouts
- 1 missing-device timeout
| Build / PR | Configuration | Helix queue | Helix machine | Android device | Cause |
|---|---|---|---|---|---|
| [#1579911](https://dev.azure.com/dnceng-public/public/_build/results?buildId=1579911&view=logs) / [#12650](https://github.com/dotnet/android/pull/12650) | android-arm/Default | `windows.11.amd64.android-armel.open` | `DNCENGWIN-141` | `27091JEGR21946` (unauthorized) | ADB unauthorized timeout |
| [#1579675](https://dev.azure.com/dnceng-public/public/_build/results?buildId=1579675&view=logs) / [#12644](https://github.com/dotnet/android/pull/12644) | android-arm/Full R2R | `windows.11.amd64.android-armel.open` | `DNCENGWIN-141` | `27091JEGR21946` (unauthorized) | ADB unauthorized timeout |
| [#1578760](https://dev.azure.com/dnceng-public/public/_build/results?buildId=1578760&view=logs) / [#12597](https://github.com/dotnet/android/pull/12597) | android-arm64/Full R2R | `windows.11.amd64.android.open` | `DNCENGWIN-135` | `26081FDF60041K` (Pixel 6) | Package manager / UID exhaustion |
| [#1578755](https://dev.azure.com/dnceng-public/public/_build/results?buildId=1578755&view=logs) / [#12641](https://github.com/dotnet/android/pull/12641) | android-arm64/Default | `windows.11.amd64.android.open` | `DNCENGWIN-036` | `01XAYV0UJJ` (Pixel 3a) | Package manager / UID exhaustion |
| [#1578133](https://dev.azure.com/dnceng-public/public/_build/results?buildId=1578133&view=logs) / [#12631](https://github.com/dotnet/android/pull/12631) | android-arm64/Default | `windows.11.amd64.android.open` | `DNCENGWIN-006` | `9CJAY1PSW4` (Pixel 3a) | Package manager / UID exhaustion |
| [#1578128](https://dev.azure.com/dnceng-public/public/_build/results?buildId=1578128&view=logs) / [#12638](https://github.com/dotnet/android/pull/12638) | android-arm64/Full R2R | `windows.11.amd64.android.open` | `DNCENGWIN-135` | `26081FDF60041K` (Pixel 6) | Package manager / UID exhaustion |
| [#1577702](https://dev.azure.com/dnceng-public/public/_build/results?buildId=1577702&view=logs) / [#12629](https://github.com/dotnet/android/pull/12629) | android-arm64/Default | `windows.11.amd64.android.open` | `DNCENGWIN-137` | `26091FDF6001Y5` (Pixel 6) | Package manager / UID exhaustion |
| [#1577702](https://dev.azure.com/dnceng-public/public/_build/results?buildId=1577702&view=logs) / [#12629](https://github.com/dotnet/android/pull/12629) | android-arm/Full R2R | `windows.11.amd64.android-armel.open` | `DNCENGWIN-141` | `27091JEGR21946` (unauthorized) | ADB unauthorized timeout |
| [#1576771](https://dev.azure.com/dnceng-public/public/_build/results?buildId=1576771&view=logs) / [#12585](https://github.com/dotnet/android/pull/12585) | android-arm64/Default | `windows.11.amd64.android.open` | `DNCENGWIN-036` | `01XAYV0UJJ` (Pixel 3a) | Package manager / UID exhaustion |
| [#1576613](https://dev.azure.com/dnceng-public/public/_build/results?buildId=1576613&view=logs) / [#12614](https://github.com/dotnet/android/pull/12614) | android-arm64/Full R2R | `windows.11.amd64.android.open` | `DNCENGWIN-034` | None detected | No Android device timeout |
| [#1576382](https://dev.azure.com/dnceng-public/public/_build/results?buildId=1576382&view=logs) / [#12541](https://github.com/dotnet/android/pull/12541) | android-arm64/Default | `windows.11.amd64.android.open` | `DNCENGWIN-020` | `9CBAY1NP1W` (Pixel 3a) | Package manager / UID exhaustion |
| [#1575529](https://dev.azure.com/dnceng-public/public/_build/results?buildId=1575529&view=logs) / [#12604](https://github.com/dotnet/android/pull/12604) | android-arm64/Full R2R | `windows.11.amd64.android.open` | `DNCENGWIN-010` | `94SAY0SM1M` (Pixel 3a) | Package manager / UID exhaustion |
The ARM32 authorization failures are concentrated on one host/device pair: `DNCENGWIN-141` / `27091JEGR21946`.
The ARM64 failures are broader. `adb uninstall` first reports:
```text
Failure [DELETE_FAILED_INTERNAL_ERROR]
```
The subsequent installation then fails with:
```text
INSTALL_FAILED_INSUFFICIENT_STORAGE: Scanning Failed.: Package ... could not be assigned a valid UID
```
This signature occurred on seven Helix machines and six Android devices (Pixel 3a and Pixel 6 models).
Same-commit reruns also show infrastructure flakiness:
- PR #12597: failed in build 1578760, MAUI task passed in 1579125
- PR #12629: failed in build 1577702, passed in 1578074
- PR #12641: failed in build 1578755, MAUI task passed in 1579094 and 1579507
### Steps to Reproduce
Run the public `dotnet-android` PR pipeline repeatedly. In the affected `MAUI Tests > MAUI Integration > Run MAUI R2R APKs on Helix` task, inspect the linked failed Helix work-item console.
The failures are intermittent and tied to the assigned Helix host/device rather than a known source change.
### Did you find any workaround?
Rerun the pipeline so that Helix schedules the work item on a healthy device.
For the device lab:
- quarantine or re-authorize Android device `27091JEGR21946` on `DNCENGWIN-141`
- clean/reprovision the affected ARM64 devices' package-manager state
- ensure a device is attached and authorized before assigning work to `DNCENGWIN-034`
### Relevant log output
ARM32 authorization failure:
```text
List of devices attached
27091JEGR21946 unauthorized transport_id:1
adb.exe wait-for-device
END OF WORK ITEM LOG: Command timed out, and was killed
```
ARM64 package-manager failure:
```text
adb.exe uninstall com.xamarin.mauir2r.arm64.fullr2r
Failure [DELETE_FAILED_INTERNAL_ERROR]
adb.exe install -r MauiR2R-android-arm64-fullr2r-Signed.apk
Failure [INSTALL_FAILED_INSUFFICIENT_STORAGE: Scanning Failed.: Package ... could not be assigned a valid UID]
```
Missing device failure:
```text
List of devices attached
adb.exe wait-for-device
END OF WORK ITEM LOG: Command timed out, and was killed
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by rerunning the public dotnet-android PR pipeline and inspect the linked Helix work-item console for the MAUI R2R task. Compare host and device assignments with the authorization, package-manager, and missing-device log signatures. Done means the affected devices are authorized and available, their package-manager state is healthy, and repeated runs pass on both Android queues.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- android
- Domain
- ci-cd, mobile-dev, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100