Helix Android queues repeatedly assign unhealthy devices to MAUI R2R work items
- Dominant language
- C#
- Stars
- 729
- Forks
- 397
- Avg merge
- 3d 15m
- Merged PRs (30d)
- 149
Description
### Summary
The `dotnet/android` public PR pipeline is seeing frequent infrastructure failures in MAUI ReadyToRun smoke tests submitted to the Helix Android device queues.
The submissions succeed and the work items are scheduled, but device setup fails before the APK is launched. This is affecting unrelated non-fork PRs and same-commit reruns pass when assigned to healthy devices.
Full dotnet/android tracking issue: https://github.com/dotnet/android/issues/12658
### Impact
Sample window: the latest 100 completed `dotnet-android` builds from 2026-08-31 through 2026-09-03.
Of 92 builds containing the MAUI R2R Helix task, 72 passed, 19 failed, and 1 was skipped. Excluding eight separately understood fork-token failures, **11 direct-PR builds contained 12 failed Helix work items**:
- 8 Android package-manager/UID exhaustion failures
- 3 ADB authorization timeouts
- 1 work item with no Android device detected
All 12 failed before APK launch and therefore do not demonstrate a MAUI or ReadyToRun product regression.
### Affected work items
| Build / PR | Configuration | Helix queue | Helix machine | Android device | Cause |
|---|---|---|---|---|---|
| [1579911](https://dev.azure.com/dnceng-public/public/_build/results?buildId=1579911&view=logs) / [dotnet/android#12650](https://github.com/dotnet/android/pull/12650) | android-arm/Default | `windows.11.amd64.android-armel.open` | `DNCENGWIN-141` | `27091JEGR21946` (unauthorized) | ADB authorization timeout |
| [1579675](https://dev.azure.com/dnceng-public/public/_build/results?buildId=1579675&view=logs) / [dotnet/android#12644](https://github.com/dotnet/android/pull/12644) | android-arm/Full R2R | `windows.11.amd64.android-armel.open` | `DNCENGWIN-141` | `27091JEGR21946` (unauthorized) | ADB authorization timeout |
| [1578760](https://dev.azure.com/dnceng-public/public/_build/results?buildId=1578760&view=logs) / [dotnet/android#12597](https://github.com/dotnet/android/pull/12597) | android-arm64/Full R2R | `windows.11.amd64.android.open` | `DNCENGWIN-135` | `26081FDF60041K` (Pixel 6) | Package manager / UID exhaustion |
| [1578755](https://dev.azure.com/dnceng-public/public/_build/results?buildId=1578755&view=logs) / [dotnet/android#12641](https://github.com/dotnet/android/pull/12641) | android-arm64/Default | `windows.11.amd64.android.open` | `DNCENGWIN-036` | `01XAYV0UJJ` (Pixel 3a) | Package manager / UID exhaustion |
| [1578133](https://dev.azure.com/dnceng-public/public/_build/results?buildId=1578133&view=logs) / [dotnet/android#12631](https://github.com/dotnet/android/pull/12631) | android-arm64/Default | `windows.11.amd64.android.open` | `DNCENGWIN-006` | `9CJAY1PSW4` (Pixel 3a) | Package manager / UID exhaustion |
| [1578128](https://dev.azure.com/dnceng-public/public/_build/results?buildId=1578128&view=logs) / [dotnet/android#12638](https://github.com/dotnet/android/pull/12638) | android-arm64/Full R2R | `windows.11.amd64.android.open` | `DNCENGWIN-135` | `26081FDF60041K` (Pixel 6) | Package manager / UID exhaustion |
| [1577702](https://dev.azure.com/dnceng-public/public/_build/results?buildId=1577702&view=logs) / [dotnet/android#12629](https://github.com/dotnet/android/pull/12629) | android-arm64/Default | `windows.11.amd64.android.open` | `DNCENGWIN-137` | `26091FDF6001Y5` (Pixel 6) | Package manager / UID exhaustion |
| [1577702](https://dev.azure.com/dnceng-public/public/_build/results?buildId=1577702&view=logs) / [dotnet/android#12629](https://github.com/dotnet/android/pull/12629) | android-arm/Full R2R | `windows.11.amd64.android-armel.open` | `DNCENGWIN-141` | `27091JEGR21946` (unauthorized) | ADB authorization timeout |
| [1576771](https://dev.azure.com/dnceng-public/public/_build/results?buildId=1576771&view=logs) / [dotnet/android#12585](https://github.com/dotnet/android/pull/12585) | android-arm64/Default | `windows.11.amd64.android.open` | `DNCENGWIN-036` | `01XAYV0UJJ` (Pixel 3a) | Package manager / UID exhaustion |
| [1576613](https://dev.azure.com/dnceng-public/public/_build/results?buildId=1576613&view=logs) / [dotnet/android#12614](https://github.com/dotnet/android/pull/12614) | android-arm64/Full R2R | `windows.11.amd64.android.open` | `DNCENGWIN-034` | None detected | No-device timeout |
| [1576382](https://dev.azure.com/dnceng-public/public/_build/results?buildId=1576382&view=logs) / [dotnet/android#12541](https://github.com/dotnet/android/pull/12541) | android-arm64/Default | `windows.11.amd64.android.open` | `DNCENGWIN-020` | `9CBAY1NP1W` (Pixel 3a) | Package manager / UID exhaustion |
| [1575529](https://dev.azure.com/dnceng-public/public/_build/results?buildId=1575529&view=logs) / [dotnet/android#12604](https://github.com/dotnet/android/pull/12604) | android-arm64/Full R2R | `windows.11.amd64.android.open` | `DNCENGWIN-010` | `94SAY0SM1M` (Pixel 3a) | Package manager / UID exhaustion |
### Failure signatures
The ARM32 failures are concentrated on one host/device pair:
```text
Helix machine: DNCENGWIN-141
Android device: 27091JEGR21946
Queue: windows.11.amd64.android-armel.open
List of devices attached
27091JEGR21946 unauthorized transport_id:1
adb.exe wait-for-device
END OF WORK ITEM LOG: Command timed out, and was killed
```
The ARM64 package-manager failures occur across seven Helix machines and six Android devices:
```text
adb.exe uninstall com.xamarin.mauir2r.arm64.fullr2r
Failure [DELETE_FAILED_INTERNAL_ERROR]
adb.exe install -r MauiR2R-android-arm64-fullr2r-Signed.apk
Failure [INSTALL_FAILED_INSUFFICIENT_STORAGE: Scanning Failed.: Package ... could not be assigned a valid UID]
```
One work item on `DNCENGWIN-034` had no device listed at all before `adb wait-for-device` timed out.
### Same-commit rerun evidence
- dotnet/android#12597: failed in build 1578760; MAUI task passed in 1579125
- dotnet/android#12629: failed in build 1577702; passed in 1578074
- dotnet/android#12641: failed in build 1578755; MAUI task passed in 1579094 and 1579507
### Action requested for Engineering Services / Helix
- [ ] Quarantine or re-authorize Android device `27091JEGR21946` attached to `DNCENGWIN-141`.
- [ ] Investigate and clean/reprovision the affected ARM64 devices' package-manager state.
- [ ] Investigate why `DNCENGWIN-034` accepted work while no Android device was visible.
- [ ] Consider queue health checks that reject hosts with missing/unauthorized devices or exhausted Android package-manager state before assigning work.
Contributor guide
Research direction
Start with the linked dotnet/android tracking issue and the affected Helix work-item logs, comparing failed runs with the listed same-commit reruns. Done means the affected devices and hosts are remediated or rejected before assignment, and the requested queue health checks prevent missing, unauthorized, or exhausted Android devices from receiving work.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- android
- Domain
- ci-cd, infrastructure, testing-qa
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100