CI: emulator 'package' service unavailable during install (bundletool 'Can't find service: package')
- Dominant language
- C#
- Stars
- 2.1k
- Forks
- 579
- Avg merge
- 1d 19h
- Merged PRs (30d)
- 252
Description
## Summary
The Android emulator on CI intermittently enters a **`system_server` restart loop** under heavy resource pressure. While `system_server` is down/restarting, its `package` binder service is deregistered, so **Google bundletool** fails at APK/AAB install time while computing the device spec:
```
[BT : 1.18.3] error : Unexpected output of 'pm list features' command: 'cmd: Can't find service: package'
error XABAS0000: com.android.tools.build.bundletool.model.exceptions.AdbOutputParseException:
Unexpected output of 'pm list features' command: 'cmd: Can't find service: package'
at com.android.tools.build.bundletool.device.DeviceFeaturesParser.parse(DeviceFeaturesParser.java:42)
at com.android.tools.build.bundletool.device.DdmlibDevice.getDeviceFeatures(DdmlibDevice.java:142)
at com.android.tools.build.bundletool.device.DeviceAnalyzer.getDeviceSpec(DeviceAnalyzer.java:75)
...
```
Surfaces at `Xamarin.Android.Common.targets(2885,3)` during `-t:Install`. Same underlying disease as #11065 (which caught it in the **test-run** phase); this is the **install** phase.
## Failing build (repro)
- **Build:** https://dev.azure.com/dnceng-public/public/_build/results?buildId=1509131
- **Pipeline:** `dotnet-android` (dnceng-public), PR-validation
- **Stage / Job:** `Package Tests` → `macOS > Tests > APKs 2`
- **Failed step:** `build Mono.Android.NET_Tests-CoreCLR` (`dotnet build -t:Install`)
## Root cause — CONFIRMED from this build's logcat
The failing lane's logcat is in the published **"Test Results - APKs .NET Release - macOS 2"** artifact at `net11.0-CoreCLR/logcat-Mono.Android.NET_Tests-CoreCLR.txt` (CoreCLR builds under the Release configuration). It captures the whole failure window and shows a **`system_server` restart loop**:
| `system_server` PID | "Entered the Android system server" | `package` service died |
|---|---|---|
| 1925 | (initial) | 00:18:52.070 |
| 2657 | 00:18:56.242 | 00:21:39.442 |
| 3493 | 00:22:01.054 | 00:25:54.690 |
| 4712 | **00:25:58.942** | — |
`system_server` died and restarted **≥3 times in ~7 minutes**. bundletool ran `pm list features` at **00:25:59.4 — ~0.5 s after the 3rd restart (pid 4712) began booting** — so `PackageManagerService` had not yet re-registered the `package` binder → `cmd: Can't find service: package`.
### Proximate trigger: the Android **Watchdog** (not a direct lmkd kill)
The fatal restart that broke our install is logged explicitly at **00:25:51.850**:
```
W Watchdog: *** WATCHDOG KILLING SYSTEM PROCESS: Blocked in handler on ui thread (android.ui)
W Watchdog: *** GOODBYE!
E AndroidRuntime: DeadSystemException: The system died
I Zygote : Process 3493 exited due to signal 9 (Killed)
```
The **Watchdog** killed `system_server` because its `android.ui` handler thread was blocked past the timeout.
### Underlying condition: severe memory/CPU pressure (lmkd active throughout)
Throughout the window the emulator is thrashing — consistent with the OOM diagnosis in #11065:
- repeated `lowmemorykiller: lmkd data connection dropped` (00:18:51, 00:21:38, 00:25:53)
- continuous `Zygote: Process … exited due to signal 9 (Killed)`
- `ActivityManager: SLOW OOM ADJ …`, `libprocessgroup: Successfully killed process cgroup …`
- `SystemServerTiming: SystemUserUnlock took to complete: 32940ms`
So the accurate chain is: **resource-starved emulator → `system_server` handler thread stalls → Watchdog kills & restarts `system_server` (repeatedly) → `package` binder transiently absent → bundletool's `pm list features` fails.**
> **Relationship to #11065:** same root disease (emulator resource starvation → `system_server` death), but a different *proximate* mechanism. Here: **Watchdog kill** (blocked `android.ui` handler). In #11065's build 1362050: **direct lmkd kill**. Two signatures, one cause.
### Why the readiness gate didn't catch it
`Start emulator` (`WaitForAndroidEmulator`) passed at 00:18:08 because `system_server` pid 1925 was briefly up and serving `package` at that instant. It was Watchdog/pressure-killed 44 s later (00:18:52), after the gate had already returned. The gate is a point-in-time check with no re-verification before install.
## Mechanism references (AOSP)
- `cmd`/`pm` print `cmd: Can't find service: package` when `servicemanager` has no `package` binder (non-blocking `checkService()` returns null). [`frameworks/native/cmds/cmd/cmd.cpp`](https://android.googlesource.com/platform/frameworks/native/+/refs/heads/main/cmds/cmd/cmd.cpp)
- `sys.boot_completed` stays `1` across the restart because it is held by `init`'s property service, separate from `system_server`. [Android memory-management docs](https://developer.android.com/topic/performance/memory-management)
- `adbd` is independent, so `adb get-state`=`device` and `transport_id` is unchanged across a restart. [ADB reference](https://developer.android.com/tools/adb)
- `init` restarts the `zygote` (`class main`) service, which re-forks `system_server`, which re-registers all binders. [`init.zygote64.rc`](https://android.googlesource.com/platform/system/core/+/refs/heads/main/rootdir/init.zygote64.rc)
- bundletool has **no retry** on this path: [`DeviceFeaturesParser.java`](https://raw.githubusercontent.com/google/bundletool/master/src/main/java/com/android/tools/build/bundletool/device/DeviceFeaturesParser.java)
- lmkd/PSI killing under critical memory pressure: [AOSP lmkd](https://source.android.com/docs/core/perf/lmkd), [lmkd README](https://android.googlesource.com/platform/system/memory/lmkd/+/refs/heads/main/README.md)
## Two independent secondary problems this exposes
1. **Same-agent retry is ineffective and destructive.** `retryCountOnTaskFailure: 3` on the build+install task (a) can't help — same broken emulator/agent, and (b) re-running the full `dotnet build -t:Install` over a dirty `obj/` produced spurious `CS0436`/`CS0108`/`CS0114` duplicate-generated-type errors on retries 2–4, **burying the real "device not ready" cause** behind compiler errors (retries 2–4 never even reached the device). Incremental-build corruption class similar to #11543.
2. **The failure is not distinguishable from a real product/test failure** — `XABAS0000` + `CS0436` look like code bugs, not infrastructure.
## Proposed direction
1. **Detect early** — before `-t:Install`, gate on **package-service readiness**, not just `sys.boot_completed`. Layered probe (weakest→strongest): `getprop sys.boot_completed`=1 → `getprop init.svc.bootanim`=`stopped` → `service check package`=`found` → `pm path android`=`package:…`. Treat `Can't find service` as "device not ready". (Aligns with #11065 Option 2.)
2. **Fail distinguishably** — classify the signatures (`Can't find service: package`, `AdbOutputParseException`, and in logcat: `WATCHDOG KILLING SYSTEM PROCESS`, `lowmemorykiller`/`lmkd`, multiple `ServiceManager: service '…' died`) as an **infrastructure** failure with a unique marker, not a build/test failure. (Overlaps with #11298.)
3. **Retry on a fresh device/agent** — the fault is the emulator/agent, so a same-agent task retry is wrong; the retry must re-provision the emulator (or re-queue on a different agent) and must not re-run the full build over a dirty `obj/`.
4. **Reduce frequency (address the starvation)** — emulator stability flags for CI (Android emulator docs / android-emulator-runner): explicit RAM (`-memory 4096` / `hw.ramSize=4096`), cold boot (`-no-snapshot`), `-gpu swiftshader` (`swiftshader_indirect` deprecated since emulator 36.4.9), `-no-boot-anim`, `-noaudio`, disable animations, and cap concurrent emulators per host (~3–4 on a ~14 GB macOS runner). The `system_server` restart loop indicates the emulator is under-resourced for the number running per host.
## Related issues
- #11065 — CI: Emulator OOM kills `system_server`, causing `Mono.Android.NET_Tests-NoAot` to fail with no results (**same root disease; test-run-phase manifestation; direct-lmkd-kill variant, build 1362050**)
- #11298 — Add crash diagnostics to device test pipeline step (**detection/classification of these exact logcat signatures**)
- #12077 — Emulator version compatibility check for system images (**same error string, different cause; ruled out here — `boot_completed` reaches 1 and recovers**)
- #11543 — BuildArchive incremental build bug (**related class of retry-over-dirty-`obj` corruption**)
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with Xamarin.Android.Common.targets around line 2885 and the WaitForAndroidEmulator readiness gate, then compare the proposed probes with #11065 and #11298. Done means install-time package-service failures are detected and marked as infrastructure issues, retries use a fresh emulator or agent, and dirty obj/ retries do not obscure the original failure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- android, azure, csharp, macos
- Domain
- ci-cd, infrastructure, mobile-dev
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100