firebase / firebase/firebase-ios-sdk

Deadlock in -[APMIdentity appInstanceID]

Open
#11,526 6 comments 1 reaction 2 assignees Claimed by @pcfba View on GitHub
api: analytics
Dominant language
C++
Stars
6.7k
Forks
1.8k
Avg merge
2d 18h
Merged PRs (30d)
75

Description

### Description

We've been investigating some strange behaviors in our app that were are hard to reproduce, but manifested when foregrounding the app and as we fired off requests to refresh our content (and also send analytics events).

Eventually we figured out it seemed like some kind of contention / deadlock issue.

Eventually we were able to catch an instance and attached the debugger, where we were able to see the stack trace. The root cause seems to be a deadlock caused by contention on the `dispatch_sync` call inside `-[APMIdentity appInstanceID]` as per this screenshot:

Screenshot 2023-07-10 at 12 38 18

In our case we were using Swift concurrency which has has a number of concurrent system threads matching the number of physical cores on the system (in my case this was 6 with an iPhone 14 Pro) servicing Swift Tasks. Each thread of these 6 was blocked on the same `dispatch_sync` call and no forward progress could be made.

It would stand to reason that this issue could also occur in other situations as well where other threads are bounded, not just when using Swift concurrency.

My understanding is that using `dispatch_sync` is tricky to get right, and may lead to deadlocks when accessed recursively, especially when the caller's context is unknown. According to `man 3 dispatch_sync` this is conceptually implemented using a `dispatch_semaphore`, which is described as unsafe to use with Swift concurrency in WWDC 2021 "Swift concurrency: Behind the scenes":
Screenshot 2023-07-10 at 16 03 37

Unfortunately it's hard to introspect the `GoogleAppMeasurement` package to determine what could be the cause of the deadlock, but we also are using AppCheck, Analytics, Auth, Firestore, Functions, Dynamic Links, and RemoteConfig, among others. I could imagine this could be some kind of case where other mutual exclusion techniques elsewhere in Firebase could trigger this scenario.

### Reproducing the issue

Reproducing this is tricky. I tried calling `Analytics.appInstanceID` in a tight loop concurrently (which was what was hanging) and wasn't able to reproduce. It seems likely that there's some other Firebase logic that has a dependency on `GoogleAppMeasurement#APMIdentity` that might be causing this issue. You can see that `FIRAppAttestProvider.state` is also blocked on something in that screenshot (unfortunately I don't have that stack available anymore), but they don't look related doing a basic inspection of `FIRAppAppAttestProvider` logic.

### Firebase SDK Version

10.10

### Xcode Version

14.3

### Installation Method

Swift Package Manager

### Firebase Product(s)

Analytics, App Check, Authentication, Crashlytics, Database, DynamicLinks, Firestore, Functions, In-App Messaging, Installations, Remote Config, Storage

### Targeted Platforms

iOS

### Relevant Log Output

_No response_

### If using Swift Package Manager, the project's Package.resolved

Expand Package.resolved snippet

```json

{
"pins" : [
{
"identity" : "abseil-cpp-binary",
"kind" : "remoteSourceControl",
"location" : "https://github.com/google/abseil-cpp-binary.git",
"state" : {
"revision" : "bfc0b6f81adc06ce5121eb23f628473638d67c5c",
"version" : "1.2022062300.0"
}
},
{
"identity" : "amplitude-ios",
"kind" : "remoteSourceControl",
"location" : "https://github.com/amplitude/Amplitude-iOS",
"state" : {
"revision" : "9e3081af2f3d45931b5404c4276c6c2e21ecd671",
"version" : "8.13.0"
}
},
{
"identity" : "analytics-connector-ios",
"kind" : "remoteSourceControl",
"location" : "https://github.com/amplitude/analytics-connector-ios.git",
"state" : {
"revision" : "b44981119e2efc0edd1d2e26831c3c4c0e0264e6",
"version" : "1.0.0"
}
},
{
"identity" : "appsflyerframework",
"kind" : "remoteSourceControl",
"location" : "https://github.com/AppsFlyerSDK/AppsFlyerFramework",
"state" : {
"revision" : "d1349c8fdd18a80a776298e4c5c3526e22feff78",
"version" : "6.8.1"
}
},
{
"identity" : "ats-sdk-ios",
"kind" : "remoteSourceControl",
"location" : "https://github.com/LiveRamp/ats-sdk-ios.git",
"state" : {
"revision" : "2983fdd06ab861f9286afb2ded727305441f6a67",
"version" : "1.4.0"
}
},
{
"identity" : "combine-schedulers",
"kind" : "remoteSourceControl",
"location" : "https://github.com/pointfreeco/combine-schedulers",
"state" : {
"revision" : "aa3e575929f2bcc5bad012bd2575eae716cbcdf7",
"version" : "0.8.0"
}
},
{
"identity" : "firebase-ios-sdk",
"kind" : "remoteSourceControl",
"location" : "https://github.com/firebase/firebase-ios-sdk.git",
"state" : {
"revision" : "5034479ca0c4f32f299677fb0ba1fe4d3e3e20b4",
"version" : "10.10.0"
}
},
{
"identity" : "googleappmeasurement",
"kind" : "remoteSourceControl",
"location" : "https://github.com/google/GoogleAppMeasurement.git",
"state" : {
"revision" : "69f5b8425dd473d2f2475bba7f88f82cd9da0e36",
"version" : "10.10.0"
}
},
{
"identity" : "googledatatransport",
"kind" : "remoteSourceControl",
"location" : "https://github.com/google/GoogleDataTransport.git",
"state" : {
"revision" : "5056b15c5acbb90cd214fe4d6138bdf5a740e5a8",
"version" : "9.2.0"
}
},
{
"identity" : "googleutilities",
"kind" : "remoteSourceControl",
"location" : "https://github.com/google/GoogleUtilities.git",
"state" : {
"revision" : "871d43135925cde39ef7421d8723ce47edfdcc39",
"version" : "7.11.1"
}
},
{
"identity" : "grdb.swift",
"kind" : "remoteSourceControl",
"location" : "https://github.com/groue/GRDB.swift.git",
"state" : {
"revision" : "dd7e7f39e8e4d7a22d258d9809a882f914690b01",
"version" : "5.26.1"
}
},
{
"identity" : "grpc-binary",
"kind" : "remoteSourceControl",
"location" : "https://github.com/google/grpc-binary.git",
"state" : {
"revision" : "f1b366129d1125be7db83247e003fc333104b569",
"version" : "1.50.2"
}
},
{
"identity" : "gtm-session-fetcher",
"kind" : "remoteSourceControl",
"location" : "https://github.com/google/gtm-session-fetcher.git",
"state" : {
"revision" : "d415594121c9e8a4f9d79cecee0965cf35e74dbd",
"version" : "3.1.1"
}
},
{
"identity" : "gzipswift",
"kind" : "remoteSourceControl",
"location" : "https://github.com/1024jp/GzipSwift.git",
"state" : {
"revision" : "7a7f17761c76a932662ab77028a4329f67d645a4",
"version" : "5.2.0"
}
},
{
"identity" : "ios-sdk",
"kind" : "remoteSourceControl",
"location" : "https://github.com/spotify/ios-sdk",
"state" : {
"revision" : "c12c8630949492439e54b6482219c34bc7d65157",
"version" : "1.2.3"
}
},
{
"identity" : "leveldb",
"kind" : "remoteSourceControl",
"location" : "https://github.com/firebase/leveldb.git",
"state" : {
"revision" : "0706abcc6b0bd9cedfbb015ba840e4a780b5159b",
"version" : "1.22.2"
}
},
{
"identity" : "libwebp-xcode",
"kind" : "remoteSourceControl",
"location" : "https://github.com/SDWebImage/libwebp-Xcode.git",
"state" : {
"revision" : "0f3bdb28a1edc5e8e43876d3835d20c601ef331f",
"version" : "1.2.3"
}
},
{
"identity" : "lottie-ios",
"kind" : "remoteSourceControl",
"location" : "https://github.com/airbnb/lottie-ios",
"state" : {
"revision" : "314537ec697719fa33a9d48210f4d06a463286d3",
"version" : "3.4.3"
}
},
{
"identity" : "nanopb",
"kind" : "remoteSourceControl",
"location" : "https://github.com/firebase/nanopb.git",
"state" : {
"revision" : "819d0a2173aff699fb8c364b6fb906f7cdb1a692",
"version" : "2.30909.0"
}
},
{
"identity" : "nimbus-ios-sdk",
"kind" : "remoteSourceControl",
"location" : "https://github.com/timehop/nimbus-ios-sdk",
"state" : {
"revision" : "becc9351d48d51b8739843ca8a59a216e6c91edf",
"version" : "2.10.0"
}
},
{
"identity" : "onesignal-xcframework",
"kind" : "remoteSourceControl",
"location" : "https://github.com/OneSignal/OneSignal-XCFramework",
"state" : {
"revision" : "b87060d4a77ab6b5e96d6655b2587d3cb05e33e5",
"version" : "3.12.3"
}
},
{
"identity" : "persister",
"kind" : "remoteSourceControl",
"location" : "https://github.com/Lickability/Persister",
"state" : {
"branch" : "main",
"revision" : "556198feaa1b37cfb54328af43dcb2709002972a"
}
},
{
"identity" : "phonenumberkit",
"kind" : "remoteSourceControl",
"location" : "https://github.com/marmelroy/PhoneNumberKit",
"state" : {
"revision" : "9b051490589ae4f2fa861287407a43a87be7226c",
"version" : "3.5.8"
}
},
{
"identity" : "promises",
"kind" : "remoteSourceControl",
"location" : "https://github.com/google/promises.git",
"state" : {
"revision" : "3e4e743631e86c8c70dbc6efdc7beaa6e90fd3bb",
"version" : "2.1.1"
}
},
{
"identity" : "sdwebimage",
"kind" : "remoteSourceControl",
"location" : "https://github.com/SDWebImage/SDWebImage.git",
"state" : {
"revision" : "4178d12a44691182c2eb79c70281b14cb73f4cbf",
"version" : "5.14.3"
}
},
{
"identity" : "sdwebimageswiftui",
"kind" : "remoteSourceControl",
"location" : "https://github.com/SDWebImage/SDWebImageSwiftUI.git",
"state" : {
"revision" : "61fefe9c284fd41ddef77d02749e88f00c305196",
"version" : "2.2.2"
}
},
{
"identity" : "sdwebimagewebpcoder",
"kind" : "remoteSourceControl",
"location" : "https://github.com/SDWebImage/SDWebImageWebPCoder.git",
"state" : {
"revision" : "4416805045174938b11edb69a86e9f7dd0699952",
"version" : "0.9.1"
}
},
{
"identity" : "sentry-cocoa",
"kind" : "remoteSourceControl",
"location" : "https://github.com/getsentry/sentry-cocoa",
"state" : {
"revision" : "d277532e1c8af813981ba01f591b15bbdd735615",
"version" : "8.8.0"
}
},
{
"identity" : "swift-case-paths",
"kind" : "remoteSourceControl",
"location" : "https://github.com/pointfreeco/swift-case-paths",
"state" : {
"revision" : "870133b7b2387df136ad301ec67b2e864b51dda1",
"version" : "0.14.0"
}
},
{
"identity" : "swift-clocks",
"kind" : "remoteSourceControl",
"location" : "https://github.com/pointfreeco/swift-clocks",
"state" : {
"revision" : "20b25ca0dd88ebfb9111ec937814ddc5a8880172",
"version" : "0.2.0"
}
},
{
"identity" : "swift-collections",
"kind" : "remoteSourceControl",
"location" : "https://github.com/apple/swift-collections",
"state" : {
"revision" : "f504716c27d2e5d4144fa4794b12129301d17729",
"version" : "1.0.3"
}
},
{
"identity" : "swift-composable-architecture",
"kind" : "remoteSourceControl",
"location" : "https://github.com/pointfreeco/swift-composable-architecture.git",
"state" : {
"revision" : "3e8eee1efe99d06e99426d421733b858b332186b",
"version" : "0.52.0"
}
},
{
"identity" : "swift-custom-dump",
"kind" : "remoteSourceControl",
"location" : "https://github.com/pointfreeco/swift-custom-dump",
"state" : {
"revision" : "de8ba65649e7ee317b9daf27dd5eebf34bd4be57",
"version" : "0.9.1"
}
},
{
"identity" : "swift-dependencies",
"kind" : "remoteSourceControl",
"location" : "https://github.com/pointfreeco/swift-dependencies",
"state" : {
"revision" : "6bb1034e8a1bfbf46dfb766b6c09b7b17e1cba10",
"version" : "0.2.0"
}
},
{
"identity" : "swift-gen",
"kind" : "remoteSourceControl",
"location" : "https://github.com/pointfreeco/swift-gen.git",
"state" : {
"revision" : "5bd20fb662e1ead7ee47df6bb0a15398295f2e06",
"version" : "0.4.0"
}
},
{
"identity" : "swift-identified-collections",
"kind" : "remoteSourceControl",
"location" : "https://github.com/pointfreeco/swift-identified-collections",
"state" : {
"revision" : "ad3932d28c2e0a009a0167089619526709ef6497",
"version" : "0.7.0"
}
},
{
"identity" : "swift-nonempty",
"kind" : "remoteSourceControl",
"location" : "https://github.com/pointfreeco/swift-nonempty.git",
"state" : {
"revision" : "8074e5f2e4a6de5a42476e545cc33438021aa2a4",
"version" : "0.4.0"
}
},
{
"identity" : "swift-overture",
"kind" : "remoteSourceControl",
"location" : "https://github.com/pointfreeco/swift-overture",
"state" : {
"revision" : "7977acd7597f413717058acc1e080731249a1d7e",
"version" : "0.5.0"
}
},
{
"identity" : "swift-package-manager-google-mobile-ads",
"kind" : "remoteSourceControl",
"location" : "https://github.com/googleads/swift-package-manager-google-mobile-ads.git",
"state" : {
"revision" : "3a03edf8728c50bc380b4a9187adf178ea477a6d",
"version" : "9.12.0"
}
},
{
"identity" : "swift-package-manager-google-user-messaging-platform",
"kind" : "remoteSourceControl",
"location" : "https://github.com/googleads/swift-package-manager-google-user-messaging-platform.git",
"state" : {
"revision" : "3b924ce3313a5fd2fc6a8dc889a8c38f76890fe3",
"version" : "2.0.0"
}
},
{
"identity" : "swift-protobuf",
"kind" : "remoteSourceControl",
"location" : "https://github.com/apple/swift-protobuf.git",
"state" : {
"revision" : "e1499bc69b9040b29184f7f2996f7bab467c1639",
"version" : "1.19.0"
}
},
{
"identity" : "swift-tagged",
"kind" : "remoteSourceControl",
"location" : "https://github.com/pointfreeco/swift-tagged",
"state" : {
"revision" : "f3f773a5e13f3c8f0ab1ce2ae6378058acefa87d",
"version" : "0.7.0"
}
},
{
"identity" : "swiftui-navigation",
"kind" : "remoteSourceControl",
"location" : "https://github.com/pointfreeco/swiftui-navigation",
"state" : {
"revision" : "0a0e1b321d70ee6a464ecfe6b0136d9eff77ebfc",
"version" : "0.7.0"
}
},
{
"identity" : "vungleadssdk-swiftpackagemanager",
"kind" : "remoteSourceControl",
"location" : "https://github.com/Vungle/VungleAdsSDK-SwiftPackageManager.git",
"state" : {
"revision" : "3ffeb9316a625e9486f850da15dff3713dd6f9b5",
"version" : "7.0.1"
}
},
{
"identity" : "xctest-dynamic-overlay",
"kind" : "remoteSourceControl",
"location" : "https://github.com/pointfreeco/xctest-dynamic-overlay.git",
"state" : {
"revision" : "62041e6016a30f56952f5d7d3f12a3fd7029e1cd",
"version" : "0.8.3"
}
}
],
"version" : 2
}

```

### If using CocoaPods, the project's Podfile.lock

Expand Podfile.lock snippet

```yml

Replace this line with the contents of your Podfile.lock!

```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.