matrix-org / matrix-org/matrix-rust-sdk

iOS: `0xdead10cc` crashes — SQLite WAL locks held by tokio blocking threads during background suspension

Open
#6,366 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
2.3k
Forks
500
Avg merge
1d 16h
Merged PRs (30d)
106

Description

## Summary

iOS terminates our app with `0xdead10cc` (RUNNINGBOARD) because the Rust SDK's tokio blocking thread pool continues holding SQLite file locks after `SyncService.stop()` returns. The app is suspended while a WAL commit (`fsync` → `sqlite3PagerCommitPhaseOne`) is still in progress on a tokio blocking thread, and iOS kills the process for holding a file lock in the suspended state.

**`SyncService.stop()` stops the sync loop, but does not wait for in-flight SQLite operations on the tokio blocking pool to complete, nor does it close the underlying database connections.**

## Environment

- **SDK version**: `matrix-rust-components-swift` 26.03.10
- **Platform**: iOS 26.3.1 / 26.4, ARM64
- **SQLite**: SQLCipher (via the SDK's embedded build)
- **Pool config**: `.poolMaxSize(poolMaxSize: 1)` — already minimized
- **Frequency**: Reproducible daily on real devices via TestFlight. 5 crashes from 2 different devices in 2 days.

## Crash Mechanism

`0xdead10cc` is an iOS-specific termination code. When an app is suspended (backgrounded), iOS checks whether any threads hold file locks (via `fcntl`, `flock`, or SQLite WAL locks). If they do, iOS sends `SIGKILL` with termination reason `RUNNINGBOARD 0xdead10cc`.

This is not a conventional crash — it cannot be caught by signal handlers or crash reporters like Sentry. It only appears in Apple's crash logs (Xcode Organizer / TestFlight).

## Definitive Stack Trace (Thread 17)

This is the thread holding the SQLite file lock at the moment of termination:

```
Thread 17 name:
Thread 17:
0 libsystem_kernel.dylib fsync + 8
1 MatrixRustSDK unixSync + 72
2 MatrixRustSDK pagerWalFrames + 1928
3 MatrixRustSDK sqlite3PagerCommitPhaseOne + 376
4 MatrixRustSDK sqlite3BtreeCommitPhaseOne + 160
5 MatrixRustSDK vdbeCommit + 1208
6 MatrixRustSDK sqlite3VdbeHalt + 1368
7 MatrixRustSDK sqlite3VdbeExec + 45388
8 MatrixRustSDK sqlite3_step + 608
9 MatrixRustSDK rusqlite::Connection::execute_batch + 276
10 MatrixRustSDK ::poll + 804
11 MatrixRustSDK tokio::runtime::task::core::Core::poll + 68
12 MatrixRustSDK tokio::runtime::task::harness::Harness::poll + 84
13 MatrixRustSDK tokio::runtime::blocking::pool::Inner::run + 416
14 MatrixRustSDK std::sys::backtrace::__rust_begin_short_backtrace + 224
```

The call chain shows:
1. A `rusqlite::Connection::execute_batch` is running on a **tokio blocking thread**
2. It is inside `sqlite3PagerCommitPhaseOne` → `pagerWalFrames` → `fsync`
3. This is a **WAL checkpoint/commit** that holds an exclusive file lock
4. iOS suspends the process and sees the lock → `0xdead10cc`

## Additional Context from Other Threads

**Thread 6** — The tokio runtime's I/O driver is still alive and polling:
```
Thread 6:
0 libsystem_kernel.dylib kevent + 8
1 MatrixRustSDK mio::poll::Poll::poll + 116
2 MatrixRustSDK tokio::runtime::io::driver::Driver::turn + 196
3 MatrixRustSDK tokio::runtime::time::Driver::park_internal + 680
4 MatrixRustSDK tokio::runtime::scheduler::current_thread::Context::park + 412
```

**Thread 16** — Another tokio blocking thread is idle but still alive:
```
Thread 16:
0 libsystem_kernel.dylib __psynch_cvwait + 8
1 libsystem_pthread.dylib _pthread_cond_wait + 1028
2 MatrixRustSDK std::sys::pal::unix::sync::condvar::Condvar::wait_timeout + 100
3 MatrixRustSDK tokio::runtime::blocking::pool::Inner::run + 220
```

**Thread 2** — SDK's `monitorCachedData` thread is sleeping (also potentially holding state):
```
Thread 2:
0 libsystem_kernel.dylib __semwait_signal + 8
1 libsystem_c.dylib nanosleep + 220
2 libsystem_c.dylib sleep + 52
3 Ping monitorCachedData + 660
```

## What We've Tried (App-Side Mitigations)

We've iterated through 3 rounds of increasingly aggressive app-side workarounds, none of which fully resolve the issue:

### Attempt 1: Fixed drain delay after `syncService.stop()`
- Added `Task.sleep(for: .seconds(2))` after `await syncService.stop()` to give in-flight SQLite ops time to finish
- **Result**: Still crashed — 2 seconds isn't enough for some WAL commits

### Attempt 2: Increased drain + close app-owned databases
- Increased drain to 5 seconds
- Added explicit close of app-owned SQLCipher databases (`messageIndexer.stop()`, `activityMentionStore.close()`) in `applicationWillResignActive`
- Added `backgroundTimeRemaining` logging
- **Result**: Still crashed — the SDK's own databases are the problem, not ours

### Attempt 3: Adaptive drain + pool size reduction + lifecycle changes
- Moved cleanup from `willResignActive` to `didEnterBackground` for more runway
- Built `SyncDrainStrategy.adaptiveBackground` that polls `backgroundTimeRemaining` and drains until 3s safety margin
- Set `.poolMaxSize(poolMaxSize: 1)` across all stores (app, auth, NSE) to minimize concurrent lock holders
- Expiration handler uses `.zero` drain (no time left, just release)
- **Result**: Still crashed — even with adaptive drain using all available background time (~27s), the SDK can start a new SQLite operation after our drain completes but before iOS suspends

### Why app-side fixes can't solve this

The fundamental problem is a race condition:
1. We call `syncService.stop()` and wait for it
2. We drain for N seconds
3. During or after our drain, the tokio runtime's blocking pool can still pick up queued tasks that do SQLite writes
4. iOS suspends the process at an arbitrary point
5. If a SQLite write is in progress at that exact moment → `0xdead10cc`

**No amount of waiting on the app side can guarantee safety**, because we cannot observe or control the Rust SDK's internal SQLite state.

## Crash Log Files

Five full crash logs are available. All show the same pattern:

| # | Device | iOS | Date | Session Duration | Thread Holding Lock |
|---|--------|-----|------|-----------------|-------------------|
| 1 | iPhone18,2 | 26.4 | Mar 27, 17:16 | ~1h 22m | Thread 2 (monitorCachedData) |
| 2 | iPhone18,2 | 26.4 | Mar 27, 18:48 | ~1h 25m | Thread 2 (monitorCachedData) |
| 3 | iPhone18,2 | 26.4 | Mar 27, 19:26 | ~34m | Thread 2 (monitorCachedData) |
| 4 | iPhone18,2 | 26.4 | Mar 28, 13:54 | ~18h 27m | Thread 2 (monitorCachedData) |
| 5 | iPhone17,4 | 26.3.1 | Mar 28, 20:26 | ~4h 16m | **Thread 17 (sqlite3_step → fsync)** |

Crash #5 is the most informative — it captures the exact SQLite call stack mid-WAL-commit.

## Related

- This affects any iOS app using `matrix-rust-sdk` with SQLite/SQLCipher in WAL mode
- Element X iOS may also be affected (same SDK)
- Apple documentation: [Understanding the exception types in a crash report](https://developer.apple.com/documentation/xcode/understanding-the-exception-types-in-a-crash-report) — see `0xdead10cc`

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing SyncService.stop() through the tokio blocking pool and the SDK's SQLite connection lifecycle, using the supplied sqlite3_step → fsync stack trace as the failure path. Determine how shutdown can wait for queued and in-flight operations and close SDK-owned connections; done means no SQLite file locks remain after stop before iOS suspension.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, sqlite
Domain
databases, mobile-dev
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.