MonoVM: unlocked robin_map read in monodroid_pinvoke_override causes intermittent SIGSEGV on .NET 10 (0% on .NET 9)
- Dominant language
- C#
- Stars
- 2.1k
- Forks
- 579
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 257
Description
### Android framework version
net10.0-android
### Affected platform version
.NET 10, Microsoft.Android.Runtime.Mono.36.android-arm64 36.1.69, MonoVM + JIT, arm64-v8a, targetSdk 36, .NET MAUI
### Description
## Summary
A native `SIGSEGV` occurs inside `tsl::robin_map::find_impl`, reached only from
`xamarin::android::PinvokeOverride::monodroid_pinvoke_override`. Disassembly of the shipped
`libmono-android.release.so` shows **two `find_impl` lookups executed before `pthread_mutex_lock` is
taken, while insertions into maps of the same type happen under that lock**. `tsl::robin_map` is not
thread-safe, and an insert that rehashes reallocates the bucket array, so a concurrent unlocked
reader dereferences freed memory.
This appears to be a regression introduced with .NET 10 — see the rate comparison below.
## Impact
Production .NET MAUI app, Google Play vitals, 28-day window:
| build | runtime | sessions affected |
|---|---|---|
| .NET 10 (current) | `Runtime.Mono.36` / 36.1.69 | **1.63%** |
| .NET 10 (previous) | `Runtime.Mono.36` / 36.1.69 | **1.51%** |
| .NET 9 | `Runtime.35` / 35.0.61 | **0.00%** |
The .NET 9 build is still in the field with roughly a thousand sessions and **zero** affected; at the
.NET 10 rate you would expect ~16. Its user population is ~70% Android 16, the same device mix as the
.NET 10 builds, so this is not an OS-version artefact.
The app-level user-perceived crash rate is **1.47%, above Play's 1.09% bad-behaviour threshold**, and
this single stack accounts for **92% of all affected sessions**.
## The crash
Two fault sites observed, both inside the same 308-byte symbol (traces in Relevant log output):
`+208` faults inside `memcmp` while comparing a key, `+28` faults inside `find_impl` itself while
loading from the bucket array. Both are what a freed or reallocated bucket array would produce,
depending on timing. The map type is `robin_map`.
## Evidence from the shipped binary
The build-id in the tombstone (`ae7ad8820dd75996ecdee692e377f8586298ceb0`) matches the pack binary
exactly, so this is analysis of the binary that actually crashed.
**Sole caller.** Scanning `.text` for AArch64 `BL` instructions targeting either `find_impl`
instantiation gives **6 call sites, all inside `monodroid_pinvoke_override`** and nowhere else.
**Lock ordering.** The call sequence (full dump in Relevant log output) is: two `find_impl` calls at
`+0x430` and `+0x45c`, *then* `pthread_mutex_lock` at `+0x490`, then further lookups plus
`operator new`, a `robin_hash` constructor and two inserts, then `pthread_mutex_unlock` at `+0xa98`.
Readers on the fast path are not excluded from the writers.
## .NET 9 comparison
The same function in `Microsoft.Android.Runtime.35` / 35.0.61 is **2,316 bytes with 56 calls and
contains no mutex operations at all**, and uses `mono_string_to_utf8` where the .NET 10 version
constructs `std::string` via `operator new`/`memcpy`. The function was evidently restructured between
the two releases, with a lock introduced covering the mutating half only.
We are not claiming .NET 9 was race-free — only that the code changed, and that the field crash rate
went from effectively zero to ~1.6% of sessions at the same time.
## What we could not determine
Binary analysis proves that maps **of the same type** are read outside the lock and mutated inside it.
It cannot prove they are the same map **instances**. If they are distinct objects — a read-only
precompiled table on the fast path and a separate mutable cache under the lock — the mechanism is
something else, though the crash location is unchanged.
We could not locate this function's source on `main` to confirm: it is not under
`src/native/mono/monodroid`, and `src/native/common/{include,runtime-base}` do not appear to contain
it either, so the tree looks to have been restructured since 36.1.69. **If this has already been
reworked on `main` — for example alongside #12010, which removed robin-map from the CoreCLR p/invoke
override — then the ask is a backport to the .NET 10 servicing branch rather than a new fix.**
Note that `Microsoft.Android.Runtime.CoreCLR.36` 36.1.69 still contains both `find_impl`
instantiations and `monodroid_pinvoke_override`, so switching runtime does not avoid this path in the
shipped .NET 10 packs.
Related: stack traces for this crash truncate at one or two frames because the .NET 10 runtime
libraries ship without `.eh_frame`. Filed separately.
### Steps to Reproduce
We have no local repro. It is intermittent and only visible at scale — roughly 1.6% of sessions
across ~16k sessions in 28 days. It occurs across every device family and on Android 13 through 17,
with no correlation to manufacturer or chipset.
The static analysis is fully reproducible from the pack binary alone, with no app required:
1. Take `Microsoft.Android.Runtime.Mono.36.android-arm64/36.1.69/runtimes/android-arm64/native/libmono-android.release.so`
2. Confirm the build-id is `ae7ad8820dd75996ecdee692e377f8586298ceb0` (`readelf -n`)
3. Disassemble `_ZN7xamarin7android15PinvokeOverride26monodroid_pinvoke_overrideEPKcS3_` at `0xac9f4`
4. Observe two calls to `robin_hash::find_impl` before the first `pthread_mutex_lock`
5. Compare against `Microsoft.Android.Runtime.35/35.0.61`, where the same function has no mutex
### Did you find any workaround?
None.
Switching to CoreCLR does not appear to help: the `Microsoft.Android.Runtime.CoreCLR.36` 36.1.69 pack
still contains both `find_impl` instantiations and `monodroid_pinvoke_override`.
The only configuration we have evidence for being unaffected is .NET 9 (0.00% of sessions over ~1k
sessions, same device mix), which is not a practical option for us.
### Relevant log output
```shell
Google Play tombstone, fault site 1:
#00 __memcmp_aarch64+12 libc.so
#01 ...robin_hash<...>::find_impl(std::string const&, unsigned long) const+208
libmono-android.release.so (BuildId: ae7ad8820dd75996ecdee692e377f8586298ceb0)
Google Play tombstone, fault site 2:
#00 ...robin_hash<...>::find_impl(std::string const&, unsigned long) const+28
libmono-android.release.so (BuildId: ae7ad8820dd75996ecdee692e377f8586298ceb0)
Call sequence of monodroid_pinvoke_override, net10 Runtime.Mono.36 / 36.1.69
(0xac9f4, 3896 bytes, 84 direct calls) — offsets from function start:
+0x430 robin_hash::find_impl <-- lookup, NO LOCK HELD
+0x45c robin_hash::find_impl <-- lookup, NO LOCK HELD
+0x490 pthread_mutex_lock@plt
+0x4b0 XXH3_64bits
+0x4c0 robin_hash::find_impl
+0x518 MonodroidDl::monodroid_dlopen
+0x604 pthread_mutex_lock@plt
+0x618 robin_hash::find_impl
+0x63c operator new
+0x65c robin_hash::robin_hash(...) <-- construct
+0x680 robin_hash::insert/emplace(...) <-- INSERT
+0x698 robin_hash::find_impl
+0x80c robin_hash::insert/emplace(...) <-- INSERT
+0xa98 pthread_mutex_unlock@plt
Same function, net9 Runtime.35 / 35.0.61 (0x4cb94, 2316 bytes, 56 direct calls):
no pthread_mutex_lock / pthread_mutex_unlock calls anywhere in the function.
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by checking whether the PinvokeOverride source has moved from src/native/mono/monodroid and src/native/common/{include,runtime-base}, then compare main with the .NET 10 servicing branch and #12010. Reproduce the disassembly against the 36.1.69 pack and determine whether the unlocked reads and locked inserts share map instances; done means a verified fix or backport addressing this crash path.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- android, cpp, csharp
- Domain
- mobile-dev
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100