firebase / firebase/firebase-android-sdk

Firestore `call was cancelled` crash occurs across multiple gRPC call paths

Open
#8,601 0 comments 22 reactions 1 assignee Claimed by @dlarocque View on GitHub
api: firestore
Dominant language
Java
Stars
2.6k
Forks
710
Avg merge
2d 9h
Merged PRs (30d)
31

Description

### [READ] Step 1: Are you in the right place?

Yes. This issue reports a bug in the Firestore code in this repository. It is not a general usage question or an application support request.

### [REQUIRED] Step 2: Describe your environment

- Android Studio version: **N/A**
- Firebase Component: **Firestore**
- Component version: **Firebase BoM 34.18.0; Firestore 26.6.0** (This crash was also observed in previous versions, including Firebase BoM 33.4.0; Firestore 25.1.2)

### [REQUIRED] Step 3: Describe the problem

#### Steps to reproduce:

The crash is observed intermittently in production. We do not have a deterministic local reproduction. The crash is reported by Crashlytics and causes the application process to terminate.

The crash appears to be the same issue reported previously [#6883](https://github.com/firebase/firebase-android-sdk/issues/6883). That issue was addressed by [#6894](https://github.com/firebase/firebase-android-sdk/pull/6894), but the crash is still occurring for users.

The crash has been observed while different gRPC call paths are being closed. `AsyncQueue` is the common execution and error-reporting layer in all three traces, so the originating application-level call cannot be determined from these stack traces. The differences appear in the underlying Firestore/gRPC calls that reach the cancellation failure.

**Observed stack traces:**

1. Firestore stops listening to a target and sends an unwatch request:
```text
Fatal Exception: java.lang.RuntimeException: Internal error in Cloud Firestore (26.6.0).
at com.google.firebase.firestore.util.AsyncQueue.lambda$panic$3(AsyncQueue.java:546)
at android.os.Handler.handleCallback(Handler.java:958)
at android.os.Handler.dispatchMessage(Handler.java:99)
at android.os.Looper.loopOnce(Looper.java:205)
at android.os.Looper.loop(Looper.java:294)
at android.app.ActivityThread.main(ActivityThread.java:8561)
at java.lang.reflect.Method.invoke(Method.java)
at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:652)
at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1026)

Caused by java.lang.RuntimeException: java.lang.IllegalStateException: call was cancelled
at com.google.firebase.firestore.util.AsyncQueue$SynchronizedShutdownAwareExecutor.lambda$executeAndReportResult$1(AsyncQueue.java:333)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:520)
at java.util.concurrent.FutureTask.run(FutureTask.java:328)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:323)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1100)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at com.google.firebase.firestore.util.AsyncQueue$SynchronizedShutdownAwareExecutor$DelayedStartFactory.run(AsyncQueue.java:235)
at java.lang.Thread.run(Thread.java:1572)

Caused by java.lang.IllegalStateException: call was cancelled
at com.google.common.base.Preconditions.checkState(Preconditions.java:513)
at io.grpc.internal.ClientCallImpl.sendMessageInternal(ClientCallImpl.java:516)
at io.grpc.internal.ClientCallImpl.sendMessage(ClientCallImpl.java:510)
at io.grpc.internal.DelayedClientCall.sendMessage(DelayedClientCall.java:340)
at io.grpc.ForwardingClientCall.sendMessage(ForwardingClientCall.java:37)
at com.google.firebase.firestore.remote.AbstractStream.writeRequest(AbstractStream.java:412)
at com.google.firebase.firestore.remote.WatchStream.unwatchTarget(WatchStream.java:99)
at com.google.firebase.firestore.remote.RemoteStore.sendUnwatchRequest(RemoteStore.java:423)
at com.google.firebase.firestore.remote.RemoteStore.stopListening(RemoteStore.java:406)
at com.google.firebase.firestore.core.SyncEngine.stopListening(SyncEngine.java:291)
at com.google.firebase.firestore.core.EventManager.removeQueryListener(EventManager.java:190)
at com.google.firebase.firestore.core.FirestoreClient.lambda$stopListening$8(FirestoreClient.java:186)
at com.google.firebase.firestore.util.AsyncQueue.lambda$enqueue$2(AsyncQueue.java:445)
at com.google.firebase.firestore.util.AsyncQueue$SynchronizedShutdownAwareExecutor.lambda$executeAndReportResult$1(AsyncQueue.java:330)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:520)
at java.util.concurrent.FutureTask.run(FutureTask.java:328)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:323)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1100)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at com.google.firebase.firestore.util.AsyncQueue$SynchronizedShutdownAwareExecutor$DelayedStartFactory.run(AsyncQueue.java:235)
at java.lang.Thread.run(Thread.java:1572)
...
```

2. Firestore closes a stream while restarting the network:
```text
Fatal Exception: java.lang.RuntimeException: Internal error in Cloud Firestore (26.3.0).
at com.google.firebase.firestore.util.AsyncQueue.lambda$panic$3(AsyncQueue.java:546)
at android.os.Handler.handleCallback(Handler.java:958)
at android.os.Handler.dispatchMessage(Handler.java:99)
at android.os.Looper.loopOnce(Looper.java:205)
at android.os.Looper.loop(Looper.java:294)
at android.app.ActivityThread.main(ActivityThread.java:8236)
at java.lang.reflect.Method.invoke(Method.java)
at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:573)
at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1058)

Caused by java.lang.IllegalStateException: call was cancelled
at com.google.common.base.Preconditions.checkState(Preconditions.java:513)
at io.grpc.internal.ClientCallImpl.halfCloseInternal(ClientCallImpl.java:500)
at io.grpc.internal.ClientCallImpl.halfClose(ClientCallImpl.java:494)
at io.grpc.internal.DelayedClientCall$7.run(DelayedClientCall.java:384)
at io.grpc.internal.DelayedClientCall.drainPendingCalls(DelayedClientCall.java:302)
at io.grpc.internal.DelayedClientCall.access$100(DelayedClientCall.java:50)
at io.grpc.internal.DelayedClientCall$1.runInContext(DelayedClientCall.java:174)
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at io.grpc.internal.ManagedChannelImpl$RealChannel$PendingCall$1.run(ManagedChannelImpl.java:1025)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:520)
at java.util.concurrent.FutureTask.run(FutureTask.java:328)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:323)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1100)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at com.google.firebase.firestore.util.AsyncQueue$SynchronizedShutdownAwareExecutor$DelayedStartFactory.run(AsyncQueue.java:235)
at java.lang.Thread.run(Thread.java:1572)
...
```

3. A pending gRPC call is started after cancellation:
```text
Fatal Exception: java.lang.RuntimeException: Internal error in Cloud Firestore (26.3.0).
at com.google.firebase.firestore.util.AsyncQueue.lambda$panic$3(AsyncQueue.java:546)
at android.os.Handler.handleCallback(Handler.java:942)
at android.os.Handler.dispatchMessage(Handler.java:99)
at android.os.Looper.loopOnce(Looper.java:240)
at android.os.Looper.loop(Looper.java:351)
at android.app.ActivityThread.main(ActivityThread.java:8377)
at java.lang.reflect.Method.invoke(Method.java)
at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:584)
at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1013)

Caused by java.lang.IllegalStateException: call was cancelled
at com.google.common.base.Preconditions.checkState(Preconditions.java:513)
at io.grpc.internal.ClientCallImpl.startInternal(ClientCallImpl.java:190)
at io.grpc.internal.ClientCallImpl.start(ClientCallImpl.java:184)
at io.grpc.internal.DelayedClientCall$2.run(DelayedClientCall.java:204)
at io.grpc.internal.DelayedClientCall.drainPendingCalls(DelayedClientCall.java:302)
at io.grpc.internal.DelayedClientCall.access$100(DelayedClientCall.java:50)
at io.grpc.internal.DelayedClientCall$1.runInContext(DelayedClientCall.java:174)
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at io.grpc.internal.ManagedChannelImpl$RealChannel$PendingCall$1.run(ManagedChannelImpl.java:1025)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:520)
at java.util.concurrent.FutureTask.run(FutureTask.java:328)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:323)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1100)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at com.google.firebase.firestore.util.AsyncQueue$SynchronizedShutdownAwareExecutor$DelayedStartFactory.run(AsyncQueue.java:235)
at java.lang.Thread.run(Thread.java:1572)
...
```

**Expected behavior:** Firestore and its gRPC call paths should safely handle cancellation, stream shutdown, and call startup, without propagating a fatal exception through `AsyncQueue.panic()`.

**Actual behavior:** A cancelled gRPC client call reaches one of several gRPC operations after cancellation: `ClientCallImpl.sendMessageInternal()`, `ClientCallImpl.halfCloseInternal()`, or `ClientCallImpl.startInternal()`. Each path throws `IllegalStateException: call was cancelled`, which is propagated through Firestore's `AsyncQueue` as an internal Firestore error and crashes the application.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.