dotnet / dotnet/android

CI: Emulator OOM kills system_server, causing Mono.Android.NET_Tests-NoAot to fail with no results

Open
#11,065 0 comments 0 reactions 0 assignees View on GitHub
flaky-ci needs-triage
Dominant language
C#
Stars
2.1k
Forks
579
Avg merge
1d 21h
Merged PRs (30d)
257

Description

## Summary

The `Mono.Android.NET_Tests-NoAot` test component fails intermittently because the Android emulator runs out of memory, triggering the Low Memory Killer (LMK) to kill `system_server`. This causes a full system restart and kills the test app before any tests execute, producing no NUnit results.

**Failing build:** https://dev.azure.com/dnceng-public/public/_build/results?buildId=1362050&view=logs&j=b227c637-c968-57be-6c5e-9409267d8fb0&t=a68687ba-0089-5801-b54b-a3b240b66a93&l=6069

**Job:** `macOS > Tests > APKs 2` in the `Package Tests` stage (`dotnet-android` pipeline on `main`)

## Timeline from logcat

| Time | Event |
|------|-------|
| 16:06:22 | `am instrument` launches test app (PID 4181) |
| 16:06:24–29 | .NET runtime initializes normally (~5s), loads assemblies, resolves `NUnitInstrumentation` |
| 16:06:29.151 | **Last log from test app** — typemap resolution completed successfully |
| 16:06:29.823 | `Process 2922 exited due to signal 9 (Killed)` — LMK begins killing processes |
| 16:06:29–32 | **~80+ system services die** (`ServiceManager: service 'xxx' died`) |
| 16:06:32.302 | `lowmemorykiller: lmkd data connection dropped` — confirms OOM |
| 16:06:32.880 | `am instrument` process shuts down with no test results |

## Errors in build log

- `warning XAAADB0000: Failure [DELETE_FAILED_INTERNAL_ERROR]` — the prior uninstall also failed, suggesting the emulator was already unhealthy
- `error: Could not find NUnit2 results file after running component Mono.Android.NET_Tests/...NUnitInstrumentation: no nunit2-results-path bundle value found`
- `warning: Unable to process TestResult-Mono.Android.NET_Tests.xml. Is it empty? (Did a unit test runner SIGSEGV?)`

## Root cause

The emulator's Low Memory Killer killed `system_server`, which cascaded into a full system restart. The test app was killed as collateral damage before it could run any tests. This is a transient infrastructure issue, not a code bug.

## Suggestions for automatic CI recovery

### Option 1: Retry the test component on failure with no results

In `TestApks.targets`, when the `RunInstrumentationTests` task detects that no `nunit2-results-path` was returned (i.e. the app produced zero output), automatically retry the component once before reporting failure. This specifically targets the "emulator died" scenario without masking real test failures.

### Option 2: Add an emulator health check before running tests

Before launching `am instrument`, run a lightweight health-check (e.g. `adb shell getprop sys.boot_completed` + `adb shell am get-config`) and if the emulator is unresponsive or recently rebooted, restart it and re-deploy before running tests. This would catch the "already unhealthy" state indicated by the `DELETE_FAILED_INTERNAL_ERROR`.

### Option 3: Use Azure DevOps automatic retry for the job

Configure the `macOS > Tests > APKs 2` job with a `retryCountOnTaskFailure` so that Azure DevOps automatically retries the entire job on failure. This is the lowest-effort option but retries the full job rather than just the failing component.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.