firebase / firebase/firebase-js-sdk
Firestore: assertion ca9 (pendingResponses < 0) still reachable in 12.17.0 after #9842, and the watch stream keeps pumping into the failed AsyncQueue (b815 flood → tab crash)
- Dominant language
- TypeScript
- Stars
- 5.1k
- Forks
- 1k
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 37
Description
### Operating System
``` Windows 10/11 (Chrome 151), Android 10 (Chrome 149), iOS 17.1 (Mobile Safari 17.1) — reproduces on all three ```
### Environment (if applicable)
``` Browser (Chrome 151.0.0.0 on Windows, Chrome 149 on Android, Mobile Safari 17.1 on iOS) ```
### Firebase SDK Version
``` 12.17.0 (compat build, loaded from https://www.gstatic.com/firebasejs/12.17.0/ — bundles @firebase/firestore 4.17.0) ```
### Firebase SDK Product(s)
Firestore
### Project Tooling
```
No bundler. Plain single-page app; compat SDK loaded via tags from gstatic.
Firestore configured as:
db.settings({ experimentalForceLongPolling: true })
no enablePersistence() — memory cache only, no IndexedDB, no multi-tab sync
App Check: ReCaptchaEnterpriseProvider, enabled before any Firestore call.
```
### Detailed Problem Description
We are hitting the assertion that #9842 was supposed to fix, on a version that already
contains that fix. Once it fires, the app is permanently dead until a full page reload,
and the tab then floods with uncaught errors until the browser kills it.
What the user sees
An open page (a dashboard with ~10 active `onSnapshot` listeners) stops updating. Every
loading placeholder stays on screen forever. The console shows one uncaught error per
second, ~985 of them before the user gives up and reloads. No listener ever fires again.
The error
```
Uncaught Error: FIRESTORE (12.17.0) INTERNAL ASSERTION FAILED: Unexpected state (ID: b815)
CONTEXT: {"rl":"Error: FIRESTORE (12.17.0) INTERNAL ASSERTION FAILED: Unexpected state
(ID: ca9) CONTEXT: {\"M\":-1,\"targetId\":1002}\n
at Rt (https://www.gstatic.com/firebasejs/12.17.0/firebase-firestore-compat.js:1:200577)
at f (…:1:200665)
at fi.J (…:1:252100)
at (…:1:252585)
at Array.forEach (<anonymous>)
at Di.forEachTarget (…:1:252975)
at Di.ae (…:1:252471)
at Cd (…:7:2746)
at Ra.onNext (…:1:283128)
at (…:1:280929)"}
at Rt (assert.ts:89:9)
at T (assert.ts:63:3)
at Bp.Xc (async_queue_impl.ts:239:7)
at Bp.enqueue (async_queue_impl.ts:120:10)
at Bp.enqueueAndForget (async_queue_impl.ts:94:10)
at persistent_stream.ts:592:18
at Na.Tt (persistent_stream.ts:531:7)
at Na.At (stream_bridge.ts:94:10)
at webchannel_connection.ts:432:26
at b.<anonymous> (webchannel_connection.ts:70:7)
```
Reading the two IDs against the source (I read the repo at commit `317dcd9`):
`0x0ca9` is the `hardAssert` in `TargetState.recordTargetResponse()`
(`packages/firestore/src/remote/watch_change.ts:247-254`):
``pendingResponses` is less than 0. This indicates that the SDK received more target acks from the server than expected. The SDK should not continue to operate.`
The reported context `{"M":-1,"targetId":1002}` is
`{ pendingResponses: -1, targetId: 1002 }`.
`0xb815` is `AsyncQueueImpl.verifyNotFailed()`
(`packages/firestore/src/util/async_queue_impl.ts:239`): `'AsyncQueue is already failed'`.
So there are two distinct problems here.
---
Problem A — `ca9` is still reachable on 12.17.0, i.e. after #9842
`402b1f0` / #9842 ("Assertion ID: ca9 (pendingResponses less than 0) caused by target
creation race condition") shipped in `@firebase/firestore@4.14.1` = `firebase@12.13.0`.
`85f6f4e` / #9985 ("improved robustness and logging in query listen stream creation and
re-creation") is also in. I verified both are ancestors of the `12.17.0` release commit
(`d459c97`). We run `12.17.0`, which bundles `@firebase/firestore@4.17.0` — so both
fixes are present and the assertion still fires.
Notably we are not in the configurations usually blamed for this class of bug:
`experimentalForceLongPolling: true` (so this is not WebChannel streaming-specific)
no IndexedDB persistence — memory cache only
no multi-tab synchronization
single tab, single `FirebaseApp`, single `Firestore` instance, one copy of the SDK
The `targetId: 1002` is worth noting: this is a long-lived page that allocates many
transient listen targets over a session (a large number of one-shot `get()` calls
alongside ~10 long-lived `onSnapshot` listeners). Frequency correlates with session
length rather than with any single user action, which is what you would expect from a
race in target allocation / re-listen rather than from one specific query.
Problem B — after the fatal assertion, the stream keeps delivering into the dead queue
This is the part that turns a single internal error into a browser-tab crash, and I think
it is independently fixable regardless of Problem A.
The `ca9` assertion kills the `AsyncQueue` (that is its stated intent: "The SDK should not
continue to operate"). But the WebChannel/watch stream is not torn down. Per the stack
above, every subsequent message still arrives at `webchannel_connection.ts` →
`stream_bridge.ts` → `persistent_stream.ts:592` → `enqueueAndForget()` →
`verifyNotFailed()` → `fail(0xb815)` — uncaught, roughly once per second, unbounded.
Consequences for an app in production:
Unbounded uncaught errors. On long-lived tabs we have seen the renderer die outright
(Chrome `STATUS_ACCESS_VIOLATION`, "Aw, Snap"), which loses unsaved user input.
There is no supported recovery. `terminate()` cannot help, because `terminate()` itself
enqueues onto the failed queue. `clearPersistence()` likewise. So a full page reload
is the only recovery — from an app's point of view, one internal assertion means the
whole Firestore client is unrecoverable for the lifetime of the document.
It is not silenceable. Because these surface as uncaught errors on `window`, any global
error reporter records hundreds of duplicates of a single fault.
Requests for Problem B:
When the `AsyncQueue` enters the failed state, close the watch/write streams (and stop
the WebChannel) instead of leaving them delivering messages into a queue that can only
throw.
And/or make `enqueueAndForget()` a no-op once the queue has failed — the "forget"
variant has no caller waiting on a result, so converting a guaranteed uncaught throw
into a single logged warning loses nothing and stops the flood.
Consider making the failed state observable to the application (e.g. surface it through
`onSnapshotsInSync` error handling or a dedicated event), so an app can tell the user
"reload to resume" instead of showing a spinner that will never resolve. Today an app
can only detect this by string-matching `"INTERNAL ASSERTION FAILED"` on `window.onerror`,
which is what we had to implement.
---
A question about `sendWatchRequest` (observation from reading the source, not a proven cause)
In `packages/firestore/src/remote/remote_store.ts:462`, `sendWatchRequest()` calls
`recordPendingTargetRequest(targetId)` first, and then may return early without ever
calling `.watch(...)`:
```ts
function sendWatchRequest(remoteStoreImpl, remoteTargetData): void {
remoteStoreImpl.watchChangeAggregator!.recordPendingTargetRequest(
remoteTargetData.targetId
);
if (resumeToken.approximateByteSize() > 0 || snapshotVersion > SnapshotVersion.min()) {
const sdkTargetId = remoteStoreImpl.targetIdMapRemoteToSdk.get(/* … */);
if (sdkTargetId === undefined) {
logDebug(LOG_TAG, 'SDK target ID not found for remote ID: ' + remoteTargetData.targetId);
// There's already a new remoteStoreListen request for the original target, so ignore this.
return; // ← pending recorded, but no watch request sent
}
// …
}
ensureWatchStream(remoteStoreImpl).watch(remoteTargetData);
}
```
On that path `pendingResponses` is incremented for a request that is never sent, so no ack
can ever balance it and `TargetState.isPending` stays `true` for that target. Is that
intended? The sign is the opposite of `ca9` (count too high, not negative), so it is
probably a different symptom — a target that stays permanently pending — but it is in the
same accounting path, and a target whose snapshot is never raised would look to an app
exactly like the "spinner that never resolves" we are chasing.
### Steps and code to reproduce issue
I do not have a minimal reproduction, and I want to be straightforward about that. The
failure is a race: it appears after a session has been open for a while (minutes to hours),
across different users, browsers and operating systems, and never on a specific user action
we can point to. What follows is the shape of the app and everything we can pin down.
Setup:
```js
firebase.initializeApp(firebaseConfig);
firebase.appCheck().activate(
new firebase.appCheck.ReCaptchaEnterpriseProvider(SITE_KEY), true);
const db = firebase.firestore();
db.settings({ experimentalForceLongPolling: true });
// deliberately no enablePersistence() — memory cache only
```
Load shape:
~10 long-lived collection listeners, attached once after sign-in and kept for the
lifetime of the page, e.g.:
```js
db.collection('global_purchases').orderBy('createdAt', 'desc').limit(400)
.onSnapshot(snap => { /* re-render */ }, err => console.warn(err));
```
Others use `.limit(500)`, `.limit(300)`, plain `.orderBy(...)`, and single-document
`db.doc(path).onSnapshot(...)`.
Alongside these, a large number of one-shot `db.doc(...).get()` / `collection().get()`
calls during boot and normal use. Over a session, allocated target IDs climb into the
high hundreds (`targetId: 1002` in the report above).
Writes are ordinary `set(..., { merge: true })` / `update()` / occasional `batch()`, some
issued while listeners are live.
Auth is a custom token; the page also signs in anonymously in some flows. Auth state can
change while listeners are attached.
What we have already ruled out on our side (each of these was tried in production over
several releases, and the assertion survived all of them):
multi-tab synchronization — disabled
IndexedDB persistence — removed entirely (memory cache only)
listener churn — global listeners are now attached exactly once and never detached until
sign-out; they are no longer torn down and re-attached on navigation
long-polling — forced
Frequency: intermittent; on a busy day, several times across our user base. When it
happens, that page is dead until reload.
If it would help, I can add temporary instrumentation around
`recordPendingTargetRequest` / `recordTargetResponse` (target ID, resume-token presence,
and whether the target was just re-listened) on a patched build and report what precedes
the negative count. Tell me what you would find most useful and I will collect it.
Contributor guide
Assessment
This issue has not been assessed yet.