envoyproxy / envoyproxy/envoy-mobile

ios: tear down earlier based on application termination

Open
#1,007 3 comments 0 reactions 2 assignees Claimed by @junr03 View on GitHub
core no stalebot platform/ios
Dominant language
Java
Stars
566
Forks
85
PR merge metrics
No merged PRs in 30d

Description

### Context

Envoy Mobile currently still has a few crashes that occur on shutdown on iOS. As far as the team is aware, these crashes are benign in the sense that users do not see the app "crash" since they happen as the OS is tearing down the application. Nonetheless, these crashes still appear in reports as crashes, and can cause noise with real issues. An example of one of these crashes is https://github.com/lyft/envoy-mobile/issues/831.

One of our theories for solving this problem is to:
- Observe the system and start to shut down as soon as we know the application is being terminated
- If possible, request that the application's lifecycle be extended by the OS

### Investigation

I did quite a bit of investigating today into what our options are for the above approaches, and arrived at the following conclusions.

It's common to watch for app lifecycle termination via `UIApplicationDelegate.applicationWillTerminate`. However, as a library, Envoy Mobile would not be able to utilize this callback. Instead, we can utilize `{NS}NotificationCenter` to observe `UIApplication.willTerminateNotification` -[ just as we do today for flushing stats](https://github.com/lyft/envoy-mobile/blob/ad1de8b01797381b902c534688848ea0ba3e2e7d/library/objective-c/EnvoyEngineImpl.m#L196-L204). Based on some experimentation, I noticed the following things:

Starting a background task when we're notified of the app terminating is too late. The OS does not respect the background task, and work is not performed when attempting to `UIApplication.beginBackgroundTask` from within `applicationWillTerminate`.

However, synchronous work does seem to complete when specified in `UIApplicationDelegate.applicationWillTerminate`. For example:

```swift
func applicationWillTerminate(_ application: UIApplication) {
NSLog("**Will terminate")
sleep(2)
NSLog("**Done sleeping in applicationWillTerminate") // This still prints to the console
}
```

Furthermore, iOS consistently waits for `applicationWillTerminate` to return before sending `NSNotification` messages with `UIApplication.willTerminateNotification`. Surprisingly, I was able to actually do more lengthy synchronous work in the `@objc` function responding to this `NSNotification`:

```swift
func applicationDidFinishLaunching(_ application: UIApplication) {
self.threadContainer.start()
NotificationCenter.default.addObserver(self, selector: #selector(self.willTerminateNotification),
name: UIApplication.willTerminateNotification, object: nil)
}

@objc private func willTerminateNotification(_: Notification) {
NSLog("**Terminating from NotificationCenter")
sleep(3)
self.threadContainer.stop()
NSLog("**Still able to log") // This still prints to the console
}
```

Finally, I actually tried running a detached thread (since this is what Envoy Mobile does) and shutting it down aggressively upon observing an `applicationWillTerminate` message from `NSNotificationCenter`:

- I was able to cancel the thread (and see `Thread.cancel()`'s function run) after doing a synchronous `sleep()` in the function responding to the termination notification
- During the `sleep()` call on the main thread, **the custom thread continued doing its work as usual**
- The thread never returned (even without canceling it - the system just kills the thread if it's not explicitly canceled)

```swift
final class CustomThread: Thread {
override func cancel() {
NSLog("**Thread canceled")
}
}

final class ThreadContainer {
private lazy var thread = CustomThread(block: {
NSLog("**Thread starting")
self.run()
NSLog("**Thread done running") // Never called
})

func start() {
NSLog("**Starting thread")
self.thread.start()
}

func stop() {
self.thread.cancel()
}

deinit {
NSLog("**Deinit") // Never called
}

func run() {
while true {
NSLog("**Running")
}
}
}

@UIApplicationMain
class AppDelegate: UIResponder, UIApplicationDelegate {
private let threadContainer = ThreadContainer()
var window: UIWindow?

func applicationDidFinishLaunching(_ application: UIApplication) {
self.threadContainer.start()
NotificationCenter.default.addObserver(self, selector: #selector(self.willTerminateNotification),
name: UIApplication.willTerminateNotification, object: nil)
}

func applicationWillTerminate(_ application: UIApplication) {
NSLog("**Will terminate")
sleep(2)
NSLog("**Done sleeping in applicationWillTerminate")
}

@objc private func willTerminateNotification(_: Notification) {
NSLog("**Terminating from NotificationCenter")
sleep(3)
self.threadContainer.stop()
NSLog("**Still able to log")
}
}
```

And this is the (trimmed down) output from the above program:

```
2020-08-06 14:50:37.059721-0700 TestBackground[878:121191] **Starting thread
2020-08-06 14:50:37.060141-0700 TestBackground[878:121438] **Thread starting
2020-08-06 14:50:37.060211-0700 TestBackground[878:121438] **Running
2020-08-06 14:50:37.060333-0700 TestBackground[878:121438] **Running
...
2020-08-06 14:50:40.368297-0700 TestBackground[878:121191] **Will terminate
2020-08-06 14:50:40.403076-0700 TestBackground[878:121438] **Running
2020-08-06 14:50:40.403440-0700 TestBackground[878:121438] **Running
...
2020-08-06 14:50:42.369505-0700 TestBackground[878:121191] **Done sleeping in applicationWillTerminate
2020-08-06 14:50:42.369724-0700 TestBackground[878:121191] **Terminating from NotificationCenter
2020-08-06 14:50:42.370548-0700 TestBackground[878:121438] **Running
2020-08-06 14:50:42.370592-0700 TestBackground[878:121438] **Running
...
2020-08-06 14:50:45.370744-0700 TestBackground[878:121191] **Thread canceled
2020-08-06 14:50:45.370793-0700 TestBackground[878:121191] **Still able to log
```

### Conclusions

1. Based on these findings, I believe we should be able to update the [existing observation to `applicationWillTerminate` in Envoy Mobile](https://github.com/lyft/envoy-mobile/blob/main/library/objective-c/EnvoyEngineImpl.m#L196-L204) to trigger a forced shutdown of Envoy Mobile's running engine.
2. **Important:** When we do this, the function reacting to `applicationWillTerminate` should not return until Envoy Mobile has finished shutting down. Based on what I saw when doing local testing (as demonstrated above), we should be able to gracefully shut down the Envoy Engine as long as we do so before returning from the triggered function.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.