dotnet / dotnet/runtime

SIGABRT "Cannot transition thread [xxxx] from RUNNING with DONE_BLOCKING" in .NET `9.0.10` on Android.

Open
#121,818 13 comments 1 reaction 1 assignee Claimed by @BrzVlad View on GitHub
area-GC-mono os-android
Dominant language
C#
Stars
18.3k
Forks
5.6k
PR merge metrics
PR metrics pending

Description

### Description

We have an Android app that has been in production for over a year. Our last release was built against .Net `9.0.9` and was stable.

After releasing an update built against .Net `9.0.10`, we started noticing large amounts of app crashes. Our investigating showed the cause to be `SIGABRT "Cannot transition thread [xxxx] from RUNNING with DONE_BLOCKING"`.

___We also sometimes see "ASYNC_SUSPEND_REQUESTED with DONE_BLOCKING".___

Our tombstones had "/bin/ping" listed in "memory near". Testing revealed triggering `System.Net.NetworkInformation.Ping` at the same time the GC was running could cause the crash to occur.

We simplified our test to call `System.Diagnostics.Process.Start` while in parallel also calling `GC.Collect()`, which reproduced the crash (https://github.com/torutek/ThreadTransitionCrash). This a contrived example to reproduce the SIGABRT as fast as possible. This is not what our app is actually doing, we do not call `GC.Collect()` manually. We test connectivity to several devices on the network using `Ping`. When the GC and ping commands happen to line up, the crash occurs.

We tried doing a single ping while calling `GC.Collect()` in parallel, and it did not crash.

It looks like `Process.Start` (or `Ping`) is causing the issue when the GC is running at the same time. We believe the PR [Don't use vfork on android](https://github.com/dotnet/runtime/pull/118331) introduced or surfaced this issue.

__This is still broken on the current version of .Net 10.__

We have `AndroidEnableSGenConcurrent` disabled in production, but the issue occurs with it `enabled` as well.

### Reproduction Steps

We created a minimal Android app (https://github.com/torutek/ThreadTransitionCrash) that reproduces the issue consistently on device or emulator.

__The .Net SDK is set to `9.0.306` (.Net `9.0.10`) in the global.json file of the repro project.__

.Net SDK `9.0.306` must be installed.

Run sample android app:

1. Clone the repro project from the provided GitHub link.
2. Install Android workloads (from directory where the repro project is cloned):
1. `dotnet workload config --update-mode manifests`
2. `dotnet workload install ios tvos macos maccatalyst maui android`
3. `dotnet workload restore`
3. Build and run the project on an Android device or emulator.
4. Allow the app to run for several seconds (~20 seconds) until it crashes with the specified error message.
* On device seems to crash faster than on emulator.
5. Observe the crash logs to confirm the presence of the "Cannot transition thread [xxxx] from RUNNING with DONE_BLOCKING" message.

### Expected behavior

Runs for several minutes without crashing.

Reproduce steps to verify .Net `9.0.9` did not have the issue (from directory where the repro project is cloned):

1. Change `global.json` workloadVersion to sdk `9.0.305`
2. Change Android workloads (from directory where the repro project is cloned):
1. `dotnet workload config --update-mode manifests`
2. `dotnet workload install ios tvos macos maccatalyst maui android`
3. `dotnet workload restore`
3. Build and run the project on an Android device or emulator.
4. Should not crash (does not crash after 10 minutes).

### Actual behavior

Crashes after several seconds with `SIGABRT "Cannot transition thread [xxxx] from RUNNING with DONE_BLOCKING".`

Snapshot of tombstone:

```
signal 6 (SIGABRT), code -1 (SI_QUEUE), fault addr --------
Abort message: 'Cannot transition thread 0x7265a84c2cf0 from RUNNING with DONE_BLOCKING'
rax 0000000000000000 rbx 00007265a84c17c8 rcx 00007268bbeb8610 rdx 0000000000000006
r8 00007265b6496ee6 r9 00007265b6496ee6 r10 00007265a84c17d0 r11 0000000000000203
r12 000072664dbd9936 r13 00007265a84c2410 r14 000000000000638d r15 00000000000063b1
rdi 000000000000638d rsi 00000000000063b1
rbp 0000000000000001 rsp 00007265a84c17c0 rip 00007268bbeb8610

15 total frames
backtrace:
#00 pc 0000000000061610 /apex/com.android.runtime/lib64/bionic/libc.so (abort+192) (BuildId: fa337969c798946280caa45e2d71a2e7)
#01 pc 0000000000054ed7 /data/app/~~AXLroOHJxVqz1V1etP9eug==/com.companyname.ThreadTransitionCrash-kUvY3gS6L-t5ToiXKGYHYg==/lib/x86_64/libmonodroid.so (xamarin::android::Helpers::abort_application(_LogCategories, char const*, bool, std::__ndk1::source_location)+135) (BuildId: f56e6bd8a5a55c1f82faaa79e186f3b95942ac52)
#02 pc 0000000000020666 /data/app/~~AXLroOHJxVqz1V1etP9eug==/com.companyname.ThreadTransitionCrash-kUvY3gS6L-t5ToiXKGYHYg==/lib/x86_64/libmonodroid.so (xamarin::android::Helpers::abort_application(char const*, bool, std::__ndk1::source_location)+22) (BuildId: f56e6bd8a5a55c1f82faaa79e186f3b95942ac52)
#03 pc 000000000002f52a /data/app/~~AXLroOHJxVqz1V1etP9eug==/com.companyname.ThreadTransitionCrash-kUvY3gS6L-t5ToiXKGYHYg==/lib/x86_64/libmonodroid.so (xamarin::android::internal::MonodroidRuntime::mono_log_handler(char const*, char const*, char const*, int, void*)+138) (BuildId: f56e6bd8a5a55c1f82faaa79e186f3b95942ac52)
#04 pc 00000000001f598b /data/app/~~AXLroOHJxVqz1V1etP9eug==/com.companyname.ThreadTransitionCrash-kUvY3gS6L-t5ToiXKGYHYg==/lib/x86_64/libmonosgen-2.0.so (BuildId: b4845e3f94890e62328505fc1f38fd33cd5a9cc1)
#05 pc 00000000001f5a3d /data/app/~~AXLroOHJxVqz1V1etP9eug==/com.companyname.ThreadTransitionCrash-kUvY3gS6L-t5ToiXKGYHYg==/lib/x86_64/libmonosgen-2.0.so (BuildId: b4845e3f94890e62328505fc1f38fd33cd5a9cc1)
#06 pc 000000000020ffdc /data/app/~~AXLroOHJxVqz1V1etP9eug==/com.companyname.ThreadTransitionCrash-kUvY3gS6L-t5ToiXKGYHYg==/lib/x86_64/libmonosgen-2.0.so (BuildId: b4845e3f94890e62328505fc1f38fd33cd5a9cc1)
#07 pc 0000000000211371 /data/app/~~AXLroOHJxVqz1V1etP9eug==/com.companyname.ThreadTransitionCrash-kUvY3gS6L-t5ToiXKGYHYg==/lib/x86_64/libmonosgen-2.0.so (BuildId: b4845e3f94890e62328505fc1f38fd33cd5a9cc1)
#08 pc 00000000001c5910 /data/app/~~AXLroOHJxVqz1V1etP9eug==/com.companyname.ThreadTransitionCrash-kUvY3gS6L-t5ToiXKGYHYg==/lib/x86_64/libmonosgen-2.0.so (BuildId: b4845e3f94890e62328505fc1f38fd33cd5a9cc1)
#09 pc 00000000001b782c /data/app/~~AXLroOHJxVqz1V1etP9eug==/com.companyname.ThreadTransitionCrash-kUvY3gS6L-t5ToiXKGYHYg==/lib/x86_64/libmonosgen-2.0.so (BuildId: b4845e3f94890e62328505fc1f38fd33cd5a9cc1)
#10 pc 00000000001b5312 /data/app/~~AXLroOHJxVqz1V1etP9eug==/com.companyname.ThreadTransitionCrash-kUvY3gS6L-t5ToiXKGYHYg==/lib/x86_64/libmonosgen-2.0.so (BuildId: b4845e3f94890e62328505fc1f38fd33cd5a9cc1)
#11 pc 000000000027d706 /data/app/~~AXLroOHJxVqz1V1etP9eug==/com.companyname.ThreadTransitionCrash-kUvY3gS6L-t5ToiXKGYHYg==/lib/x86_64/libmonosgen-2.0.so (mono_runtime_invoke_checked+134) (BuildId: b4845e3f94890e62328505fc1f38fd33cd5a9cc1)
#12 pc 0000000000293b38 /data/app/~~AXLroOHJxVqz1V1etP9eug==/com.companyname.ThreadTransitionCrash-kUvY3gS6L-t5ToiXKGYHYg==/lib/x86_64/libmonosgen-2.0.so (BuildId: b4845e3f94890e62328505fc1f38fd33cd5a9cc1)
#13 pc 00000000000cd06a /apex/com.android.runtime/lib64/bionic/libc.so (__pthread_start(void*)+58) (BuildId: fa337969c798946280caa45e2d71a2e7)
#14 pc 0000000000062d88 /apex/com.android.runtime/lib64/bionic/libc.so (__start_thread+56) (BuildId: fa337969c798946280caa45e2d71a2e7)
```

### Regression?

Bug seems to have been introduced in .Net `9.0.10`. Possibly by [Don't use vfork on android](https://github.com/dotnet/runtime/pull/118331), we are unsure if it caused the issue, or exposed it.

### Known Workarounds

Revert to .Net `9.0.9` until the issue is resolved.

### Configuration

_No response_

### Other information

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.