[Firestore] Snapshot listeners stall under concurrent load after 26.0.2 "performance improvements"

# Description

Cloud Firestore Android SDK 26.0.2 (BoM 34.4.0) introduced a severe performance regression where **snapshot listeners stall or deliver with extreme latency** under concurrent load. Downgrading to 26.0.0 (BoM 34.1.0) immediately resolves the issue.

The regression was introduced by the changes in 26.0.1–26.0.2 labeled as "performance improvements":

* #7376 — Replaced the deprecated AsyncTask thread pool with a self-managed thread pool
* #7370 — Internal memoization of calculated document data
* #7388 — Avoiding excessive Comparator instance creation
* #7389 — Using unsorted HashMap instead of sorted TreeMap

# Observed behavior

* First snapshot delivery per listener works normally (86–261ms)
* Subsequent snapshot deliveries stall for 5–60+ seconds or never arrive
* Dead silence periods where WatchStream is subscribed but no callbacks fire
* Cache loads that normally take 80–680ms degrade to 5+ seconds

# Expected behavior
Snapshot listeners should deliver consistently regardless of how many concurrent listeners are active, as they do on 26.0.0.

# Conditions that trigger the regression
These are all common, supported Firestore usage patterns:

1. 6–10 concurrent snapshot listeners (typical for tabbed UIs with dashboard widgets and detail editors)
1. Documents with 50+ fields, deeply nested maps, arrays of reference IDs, and small inline base64 strings (~3 KB)
1. Complex queries using Filter.or with multiple branches and whereIn filters
1. Background CPU work is optional — the issue reproduces without it, but concurrent CPU load (e.g. video encoding) makes it worse

# Reproduction
We've created a minimal Flutter reproduction app that uses an automated "canary writer" to measure snapshot delivery latency:

**Repository**: https://github.com/s2nventures/firestore-perf-rgn

## Steps:

1. Seed 30 large documents into a `perf_test` collection
1. Open 8 concurrent snapshot listeners with `Filter.or`, `whereIn`, equality, and nested field queries
1. Start the canary — writes to a canary document every 3 seconds and measures how long the snapshot takes to arrive (10s timeout = "missed")
1. Observe latency in the log output

### Results on 26.0.2 (BoM 34.4.0):

```
[13:23:45.741] Seeding 30 documents (~3 KB blob each)...
[13:23:48.474] Seeded 30 docs in 2688 ms
[13:23:51.372] Opening 8 concurrent snapshot listeners...
[13:23:51.386] 8 listeners active
[13:23:51.549] SNAPSHOT #1 [tenant=tenant_0]: 10 docs (cache), 10 changes
[13:23:51.580] SNAPSHOT #2 [tenant=tenant_1]: 10 docs (cache), 10 changes
[13:23:51.619] SNAPSHOT #3 [reviewer=user_0]: 15 docs (cache), 15 changes
[13:23:51.679] SNAPSHOT #4 [reviewer=user_1]: 15 docs (cache), 15 changes
[13:23:51.696] SNAPSHOT #5 [statuses=[0, 1, 2]]: 15 docs (cache), 15 changes
[13:23:51.712] SNAPSHOT #6 [statuses=[3, 4, 5]]: 15 docs (cache), 15 changes
[13:23:51.752] SNAPSHOT #7 [top-level-or]: 20 docs (cache), 20 changes
[13:23:51.753] SNAPSHOT #8 [children]: 0 docs (cache), 0 changes
[13:23:53.646] CANARY WRITE #1
[13:23:53.647] Canary started — writing every 3s, timeout 10s
[13:23:56.647] CANARY WRITE #2
[13:23:59.646] CANARY WRITE #3
[13:24:02.646] CANARY WRITE #4
[13:24:03.649] CANARY MISSED #1 — no snapshot after 10003ms (total missed: 1)
[13:24:05.646] CANARY WRITE #5
[13:24:06.648] CANARY MISSED #2 — no snapshot after 10001ms (total missed: 2)
[13:24:08.647] CANARY WRITE #6
[13:24:09.648] CANARY MISSED #3 — no snapshot after 10002ms (total missed: 3)
[13:24:11.645] CANARY WRITE #7
[13:24:12.648] CANARY MISSED #4 — no snapshot after 10001ms (total missed: 4)
[13:24:14.647] CANARY WRITE #8
[13:24:15.648] CANARY MISSED #5 — no snapshot after 10002ms (total missed: 5)
[13:24:17.646] CANARY WRITE #9
[13:24:18.649] CANARY MISSED #6 — no snapshot after 10001ms (total missed: 6)
[13:24:20.646] CANARY WRITE #10
[13:24:21.648] CANARY MISSED #7 — no snapshot after 10002ms (total missed: 7)
[13:24:23.646] CANARY WRITE #11
[13:24:25.647] CANARY MISSED #8 — no snapshot after 10999ms (total missed: 8)
[13:24:26.647] CANARY WRITE #12
[13:24:27.649] CANARY MISSED #9 — no snapshot after 10003ms (total missed: 9)
[13:24:29.646] CANARY WRITE #13
[13:24:31.558] CANARY SUMMARY: 13 writes, 0 received, 9 missed, avg: 0ms, max: 0ms
[13:24:31.558] Canary stopped
```

### Results on 26.0.0 (BoM 34.1.0):

```
[13:29:41.697] Seeding 30 documents (~3 KB blob each)...
[13:29:44.259] Seeded 30 docs in 2522 ms
[13:29:45.681] Opening 8 concurrent snapshot listeners...
[13:29:45.699] 8 listeners active
[13:29:45.855] SNAPSHOT #1 [tenant=tenant_0]: 10 docs (cache), 10 changes
[13:29:45.889] SNAPSHOT #2 [tenant=tenant_1]: 10 docs (cache), 10 changes
[13:29:45.958] SNAPSHOT #3 [reviewer=user_0]: 15 docs (cache), 15 changes
[13:29:45.978] SNAPSHOT #4 [reviewer=user_1]: 15 docs (cache), 15 changes
[13:29:45.997] SNAPSHOT #5 [statuses=[0, 1, 2]]: 15 docs (cache), 15 changes
[13:29:46.034] SNAPSHOT #6 [statuses=[3, 4, 5]]: 15 docs (cache), 15 changes
[13:29:46.060] SNAPSHOT #7 [top-level-or]: 20 docs (cache), 20 changes
[13:29:46.061] SNAPSHOT #8 [children]: 0 docs (cache), 0 changes
[13:29:51.874] CANARY WRITE #1
[13:29:51.875] Canary started — writing every 3s, timeout 10s
[13:29:51.909] CANARY #1: 35ms (avg: 35ms, max: 35ms, missed: 0)
[13:29:54.877] CANARY WRITE #2
[13:29:54.958] CANARY #2: 80ms (avg: 57ms, max: 80ms, missed: 0)
[13:29:57.879] CANARY WRITE #3
[13:29:57.961] CANARY #3: 82ms (avg: 65ms, max: 82ms, missed: 0)
[13:30:00.876] CANARY WRITE #4
[13:30:00.949] CANARY #4: 73ms (avg: 67ms, max: 82ms, missed: 0)
[13:30:03.877] CANARY WRITE #5
[13:30:03.954] CANARY #5: 76ms (avg: 69ms, max: 82ms, missed: 0)
[13:30:06.879] CANARY WRITE #6
[13:30:06.972] CANARY #6: 93ms (avg: 73ms, max: 93ms, missed: 0)
[13:30:09.879] CANARY WRITE #7
[13:30:09.973] CANARY #7: 93ms (avg: 76ms, max: 93ms, missed: 0)
[13:30:12.877] CANARY WRITE #8
[13:30:12.985] CANARY #8: 108ms (avg: 80ms, max: 108ms, missed: 0)
[13:30:15.878] CANARY WRITE #9
[13:30:15.956] CANARY #9: 77ms (avg: 79ms, max: 108ms, missed: 0)
[13:30:18.876] CANARY WRITE #10
[13:30:18.943] CANARY #10: 67ms (avg: 78ms, max: 108ms, missed: 0)
[13:30:21.877] CANARY WRITE #11
[13:30:21.954] CANARY #11: 77ms (avg: 78ms, max: 108ms, missed: 0)
[13:30:24.877] CANARY WRITE #12
[13:30:24.945] CANARY #12: 68ms (avg: 77ms, max: 108ms, missed: 0)
[13:30:27.878] CANARY WRITE #13
[13:30:27.963] CANARY #13: 84ms (avg: 77ms, max: 108ms, missed: 0)
[13:30:30.876] CANARY WRITE #14
[13:30:30.947] CANARY #14: 70ms (avg: 77ms, max: 108ms, missed: 0)
[13:30:33.879] CANARY WRITE #15
[13:30:33.959] CANARY #15: 80ms (avg: 77ms, max: 108ms, missed: 0)
[13:30:34.466] CANARY SUMMARY: 15 writes, 15 received, 0 missed, avg: 77ms, max: 108ms
[13:30:34.467] Canary stopped
```

# Environment

* Working: Cloud Firestore 26.0.0 (BoM 34.1.0)
* Broken: Cloud Firestore 26.0.2 (BoM 34.4.0)
* Platform: Android (tested on physical devices and emulators)
* Client: Flutter 3.32+, Dart 3.11+

# Tooling

This regression was identified and isolated using **Claude Opus 4.6 Thinking** (Anthropic) via the Windsurf Cascade plugin in IntelliJ.

Claude assisted with:

- Diffing the releases of our production app to identify candidate changes
- Systematically reverting changes one at a time to isolate the root cause
- Tracing the Flutter `cloud_firestore` version bump to the underlying native Android SDK BoM and changelog
- Developing a minimal reproduction app

All testing and verification were performed by the developer on physical hardware.

# Suspected root cause

The thread pool replacement in #7376 is the most likely culprit. The AsyncTask thread pool was well-integrated with Android's threading model. The replacement self-managed thread pool may have different scheduling characteristics that cause thread starvation or priority inversion when:

* Multiple Firestore query listeners compete for the same pool
* Large document deserialization blocks pool threads
* Background CPU work causes OS scheduler contention