dotnet / dotnet/android

CI: MAUI R2R Helix device-lab failures across Android queues

Open
#12,658 0 comments 0 reactions 0 assignees View on GitHub
flaky-ci needs-triage
Dominant language
C#
Stars
2.1k
Forks
579
Avg merge
1d 21h
Merged PRs (30d)
257

Description

### Android framework version

.NET 11 / `net11.0-android` CI builds.

### Affected platform version

Helix queues:

- `windows.11.amd64.android.open` (Android ARM64 devices)
- `windows.11.amd64.android-armel.open` (Android ARM32 devices)

Sample window: the latest 100 completed public `dotnet-android` pipeline builds at the time of investigation, from 2026-08-31 through 2026-09-03.

### Description

Recent non-fork PR builds are frequently failing in `Run MAUI R2R APKs on Helix`. These failures are not Helix submission/authentication failures and do not demonstrate MAUI or ReadyToRun regressions: each job was submitted and scheduled successfully, then failed on device setup before the APK launch and `pidof` smoke check.

Of 92 sampled builds containing the task, 72 passed, 19 failed, and 1 was skipped. Eight of the failures were the separately understood fork-token problem. The remaining 11 direct-PR builds contained 12 failed work items:

- 8 Android package-manager/UID exhaustion failures
- 3 ADB authorization timeouts
- 1 missing-device timeout

| Build / PR | Configuration | Helix queue | Helix machine | Android device | Cause |
|---|---|---|---|---|---|
| [#1579911](https://dev.azure.com/dnceng-public/public/_build/results?buildId=1579911&view=logs) / [#12650](https://github.com/dotnet/android/pull/12650) | android-arm/Default | `windows.11.amd64.android-armel.open` | `DNCENGWIN-141` | `27091JEGR21946` (unauthorized) | ADB unauthorized timeout |
| [#1579675](https://dev.azure.com/dnceng-public/public/_build/results?buildId=1579675&view=logs) / [#12644](https://github.com/dotnet/android/pull/12644) | android-arm/Full R2R | `windows.11.amd64.android-armel.open` | `DNCENGWIN-141` | `27091JEGR21946` (unauthorized) | ADB unauthorized timeout |
| [#1578760](https://dev.azure.com/dnceng-public/public/_build/results?buildId=1578760&view=logs) / [#12597](https://github.com/dotnet/android/pull/12597) | android-arm64/Full R2R | `windows.11.amd64.android.open` | `DNCENGWIN-135` | `26081FDF60041K` (Pixel 6) | Package manager / UID exhaustion |
| [#1578755](https://dev.azure.com/dnceng-public/public/_build/results?buildId=1578755&view=logs) / [#12641](https://github.com/dotnet/android/pull/12641) | android-arm64/Default | `windows.11.amd64.android.open` | `DNCENGWIN-036` | `01XAYV0UJJ` (Pixel 3a) | Package manager / UID exhaustion |
| [#1578133](https://dev.azure.com/dnceng-public/public/_build/results?buildId=1578133&view=logs) / [#12631](https://github.com/dotnet/android/pull/12631) | android-arm64/Default | `windows.11.amd64.android.open` | `DNCENGWIN-006` | `9CJAY1PSW4` (Pixel 3a) | Package manager / UID exhaustion |
| [#1578128](https://dev.azure.com/dnceng-public/public/_build/results?buildId=1578128&view=logs) / [#12638](https://github.com/dotnet/android/pull/12638) | android-arm64/Full R2R | `windows.11.amd64.android.open` | `DNCENGWIN-135` | `26081FDF60041K` (Pixel 6) | Package manager / UID exhaustion |
| [#1577702](https://dev.azure.com/dnceng-public/public/_build/results?buildId=1577702&view=logs) / [#12629](https://github.com/dotnet/android/pull/12629) | android-arm64/Default | `windows.11.amd64.android.open` | `DNCENGWIN-137` | `26091FDF6001Y5` (Pixel 6) | Package manager / UID exhaustion |
| [#1577702](https://dev.azure.com/dnceng-public/public/_build/results?buildId=1577702&view=logs) / [#12629](https://github.com/dotnet/android/pull/12629) | android-arm/Full R2R | `windows.11.amd64.android-armel.open` | `DNCENGWIN-141` | `27091JEGR21946` (unauthorized) | ADB unauthorized timeout |
| [#1576771](https://dev.azure.com/dnceng-public/public/_build/results?buildId=1576771&view=logs) / [#12585](https://github.com/dotnet/android/pull/12585) | android-arm64/Default | `windows.11.amd64.android.open` | `DNCENGWIN-036` | `01XAYV0UJJ` (Pixel 3a) | Package manager / UID exhaustion |
| [#1576613](https://dev.azure.com/dnceng-public/public/_build/results?buildId=1576613&view=logs) / [#12614](https://github.com/dotnet/android/pull/12614) | android-arm64/Full R2R | `windows.11.amd64.android.open` | `DNCENGWIN-034` | None detected | No Android device timeout |
| [#1576382](https://dev.azure.com/dnceng-public/public/_build/results?buildId=1576382&view=logs) / [#12541](https://github.com/dotnet/android/pull/12541) | android-arm64/Default | `windows.11.amd64.android.open` | `DNCENGWIN-020` | `9CBAY1NP1W` (Pixel 3a) | Package manager / UID exhaustion |
| [#1575529](https://dev.azure.com/dnceng-public/public/_build/results?buildId=1575529&view=logs) / [#12604](https://github.com/dotnet/android/pull/12604) | android-arm64/Full R2R | `windows.11.amd64.android.open` | `DNCENGWIN-010` | `94SAY0SM1M` (Pixel 3a) | Package manager / UID exhaustion |

The ARM32 authorization failures are concentrated on one host/device pair: `DNCENGWIN-141` / `27091JEGR21946`.

The ARM64 failures are broader. `adb uninstall` first reports:

```text
Failure [DELETE_FAILED_INTERNAL_ERROR]
```

The subsequent installation then fails with:

```text
INSTALL_FAILED_INSUFFICIENT_STORAGE: Scanning Failed.: Package ... could not be assigned a valid UID
```

This signature occurred on seven Helix machines and six Android devices (Pixel 3a and Pixel 6 models).

Same-commit reruns also show infrastructure flakiness:

- PR #12597: failed in build 1578760, MAUI task passed in 1579125
- PR #12629: failed in build 1577702, passed in 1578074
- PR #12641: failed in build 1578755, MAUI task passed in 1579094 and 1579507

### Steps to Reproduce

Run the public `dotnet-android` PR pipeline repeatedly. In the affected `MAUI Tests > MAUI Integration > Run MAUI R2R APKs on Helix` task, inspect the linked failed Helix work-item console.

The failures are intermittent and tied to the assigned Helix host/device rather than a known source change.

### Did you find any workaround?

Rerun the pipeline so that Helix schedules the work item on a healthy device.

For the device lab:

- quarantine or re-authorize Android device `27091JEGR21946` on `DNCENGWIN-141`
- clean/reprovision the affected ARM64 devices' package-manager state
- ensure a device is attached and authorized before assigning work to `DNCENGWIN-034`

### Relevant log output

ARM32 authorization failure:

```text
List of devices attached
27091JEGR21946 unauthorized transport_id:1
adb.exe wait-for-device
END OF WORK ITEM LOG: Command timed out, and was killed
```

ARM64 package-manager failure:

```text
adb.exe uninstall com.xamarin.mauir2r.arm64.fullr2r
Failure [DELETE_FAILED_INTERNAL_ERROR]

adb.exe install -r MauiR2R-android-arm64-fullr2r-Signed.apk
Failure [INSTALL_FAILED_INSUFFICIENT_STORAGE: Scanning Failed.: Package ... could not be assigned a valid UID]
```

Missing device failure:

```text
List of devices attached

adb.exe wait-for-device
END OF WORK ITEM LOG: Command timed out, and was killed
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by rerunning the public dotnet-android PR pipeline and inspect the linked Helix work-item console for the MAUI R2R task. Compare host and device assignments with the authorization, package-manager, and missing-device log signatures. Done means the affected devices are authorized and available, their package-manager state is healthy, and repeated runs pass on both Android queues.

Written by the indexing model from the issue text.

Assessment

Tech stack
android
Domain
ci-cd, mobile-dev, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.