firebase / firebase/firebase-android-sdk
Firestore `call was cancelled` crash occurs across multiple gRPC call paths
- Dominant language
- Java
- Stars
- 2.6k
- Forks
- 710
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 31
Description
### [READ] Step 1: Are you in the right place?
Yes. This issue reports a bug in the Firestore code in this repository. It is not a general usage question or an application support request.
### [REQUIRED] Step 2: Describe your environment
- Android Studio version: **N/A**
- Firebase Component: **Firestore**
- Component version: **Firebase BoM 34.18.0; Firestore 26.6.0** (This crash was also observed in previous versions, including Firebase BoM 33.4.0; Firestore 25.1.2)
### [REQUIRED] Step 3: Describe the problem
#### Steps to reproduce:
The crash is observed intermittently in production. We do not have a deterministic local reproduction. The crash is reported by Crashlytics and causes the application process to terminate.
The crash appears to be the same issue reported previously [#6883](https://github.com/firebase/firebase-android-sdk/issues/6883). That issue was addressed by [#6894](https://github.com/firebase/firebase-android-sdk/pull/6894), but the crash is still occurring for users.
The crash has been observed while different gRPC call paths are being closed. `AsyncQueue` is the common execution and error-reporting layer in all three traces, so the originating application-level call cannot be determined from these stack traces. The differences appear in the underlying Firestore/gRPC calls that reach the cancellation failure.
**Observed stack traces:**
1. Firestore stops listening to a target and sends an unwatch request:
```text
Fatal Exception: java.lang.RuntimeException: Internal error in Cloud Firestore (26.6.0).
at com.google.firebase.firestore.util.AsyncQueue.lambda$panic$3(AsyncQueue.java:546)
at android.os.Handler.handleCallback(Handler.java:958)
at android.os.Handler.dispatchMessage(Handler.java:99)
at android.os.Looper.loopOnce(Looper.java:205)
at android.os.Looper.loop(Looper.java:294)
at android.app.ActivityThread.main(ActivityThread.java:8561)
at java.lang.reflect.Method.invoke(Method.java)
at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:652)
at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1026)
Caused by java.lang.RuntimeException: java.lang.IllegalStateException: call was cancelled
at com.google.firebase.firestore.util.AsyncQueue$SynchronizedShutdownAwareExecutor.lambda$executeAndReportResult$1(AsyncQueue.java:333)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:520)
at java.util.concurrent.FutureTask.run(FutureTask.java:328)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:323)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1100)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at com.google.firebase.firestore.util.AsyncQueue$SynchronizedShutdownAwareExecutor$DelayedStartFactory.run(AsyncQueue.java:235)
at java.lang.Thread.run(Thread.java:1572)
Caused by java.lang.IllegalStateException: call was cancelled
at com.google.common.base.Preconditions.checkState(Preconditions.java:513)
at io.grpc.internal.ClientCallImpl.sendMessageInternal(ClientCallImpl.java:516)
at io.grpc.internal.ClientCallImpl.sendMessage(ClientCallImpl.java:510)
at io.grpc.internal.DelayedClientCall.sendMessage(DelayedClientCall.java:340)
at io.grpc.ForwardingClientCall.sendMessage(ForwardingClientCall.java:37)
at com.google.firebase.firestore.remote.AbstractStream.writeRequest(AbstractStream.java:412)
at com.google.firebase.firestore.remote.WatchStream.unwatchTarget(WatchStream.java:99)
at com.google.firebase.firestore.remote.RemoteStore.sendUnwatchRequest(RemoteStore.java:423)
at com.google.firebase.firestore.remote.RemoteStore.stopListening(RemoteStore.java:406)
at com.google.firebase.firestore.core.SyncEngine.stopListening(SyncEngine.java:291)
at com.google.firebase.firestore.core.EventManager.removeQueryListener(EventManager.java:190)
at com.google.firebase.firestore.core.FirestoreClient.lambda$stopListening$8(FirestoreClient.java:186)
at com.google.firebase.firestore.util.AsyncQueue.lambda$enqueue$2(AsyncQueue.java:445)
at com.google.firebase.firestore.util.AsyncQueue$SynchronizedShutdownAwareExecutor.lambda$executeAndReportResult$1(AsyncQueue.java:330)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:520)
at java.util.concurrent.FutureTask.run(FutureTask.java:328)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:323)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1100)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at com.google.firebase.firestore.util.AsyncQueue$SynchronizedShutdownAwareExecutor$DelayedStartFactory.run(AsyncQueue.java:235)
at java.lang.Thread.run(Thread.java:1572)
...
```
2. Firestore closes a stream while restarting the network:
```text
Fatal Exception: java.lang.RuntimeException: Internal error in Cloud Firestore (26.3.0).
at com.google.firebase.firestore.util.AsyncQueue.lambda$panic$3(AsyncQueue.java:546)
at android.os.Handler.handleCallback(Handler.java:958)
at android.os.Handler.dispatchMessage(Handler.java:99)
at android.os.Looper.loopOnce(Looper.java:205)
at android.os.Looper.loop(Looper.java:294)
at android.app.ActivityThread.main(ActivityThread.java:8236)
at java.lang.reflect.Method.invoke(Method.java)
at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:573)
at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1058)
Caused by java.lang.IllegalStateException: call was cancelled
at com.google.common.base.Preconditions.checkState(Preconditions.java:513)
at io.grpc.internal.ClientCallImpl.halfCloseInternal(ClientCallImpl.java:500)
at io.grpc.internal.ClientCallImpl.halfClose(ClientCallImpl.java:494)
at io.grpc.internal.DelayedClientCall$7.run(DelayedClientCall.java:384)
at io.grpc.internal.DelayedClientCall.drainPendingCalls(DelayedClientCall.java:302)
at io.grpc.internal.DelayedClientCall.access$100(DelayedClientCall.java:50)
at io.grpc.internal.DelayedClientCall$1.runInContext(DelayedClientCall.java:174)
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at io.grpc.internal.ManagedChannelImpl$RealChannel$PendingCall$1.run(ManagedChannelImpl.java:1025)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:520)
at java.util.concurrent.FutureTask.run(FutureTask.java:328)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:323)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1100)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at com.google.firebase.firestore.util.AsyncQueue$SynchronizedShutdownAwareExecutor$DelayedStartFactory.run(AsyncQueue.java:235)
at java.lang.Thread.run(Thread.java:1572)
...
```
3. A pending gRPC call is started after cancellation:
```text
Fatal Exception: java.lang.RuntimeException: Internal error in Cloud Firestore (26.3.0).
at com.google.firebase.firestore.util.AsyncQueue.lambda$panic$3(AsyncQueue.java:546)
at android.os.Handler.handleCallback(Handler.java:942)
at android.os.Handler.dispatchMessage(Handler.java:99)
at android.os.Looper.loopOnce(Looper.java:240)
at android.os.Looper.loop(Looper.java:351)
at android.app.ActivityThread.main(ActivityThread.java:8377)
at java.lang.reflect.Method.invoke(Method.java)
at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:584)
at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1013)
Caused by java.lang.IllegalStateException: call was cancelled
at com.google.common.base.Preconditions.checkState(Preconditions.java:513)
at io.grpc.internal.ClientCallImpl.startInternal(ClientCallImpl.java:190)
at io.grpc.internal.ClientCallImpl.start(ClientCallImpl.java:184)
at io.grpc.internal.DelayedClientCall$2.run(DelayedClientCall.java:204)
at io.grpc.internal.DelayedClientCall.drainPendingCalls(DelayedClientCall.java:302)
at io.grpc.internal.DelayedClientCall.access$100(DelayedClientCall.java:50)
at io.grpc.internal.DelayedClientCall$1.runInContext(DelayedClientCall.java:174)
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at io.grpc.internal.ManagedChannelImpl$RealChannel$PendingCall$1.run(ManagedChannelImpl.java:1025)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:520)
at java.util.concurrent.FutureTask.run(FutureTask.java:328)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:323)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1100)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at com.google.firebase.firestore.util.AsyncQueue$SynchronizedShutdownAwareExecutor$DelayedStartFactory.run(AsyncQueue.java:235)
at java.lang.Thread.run(Thread.java:1572)
...
```
**Expected behavior:** Firestore and its gRPC call paths should safely handle cancellation, stream shutdown, and call startup, without propagating a fatal exception through `AsyncQueue.panic()`.
**Actual behavior:** A cancelled gRPC client call reaches one of several gRPC operations after cancellation: `ClientCallImpl.sendMessageInternal()`, `ClientCallImpl.halfCloseInternal()`, or `ClientCallImpl.startInternal()`. Each path throws `IllegalStateException: call was cancelled`, which is propagated through Firestore's `AsyncQueue` as an internal Firestore error and crashes the application.
Contributor guide
Assessment
This issue has not been assessed yet.