firebase / firebase/extensions
fix(bigquery-firestore-export): soft failures throw and retry-storm
- Dominant language
- TypeScript
- Stars
- 979
- Forks
- 433
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 125
Description
Originally filed as "soft failures throw and retry-storm" from the #2974 parity analysis (bigquery-firestore-export §7a-7c, §8). The state has moved, so reframing.
## Current state
- `processMessages` now runs with `retry: false` (6a9d53dc), matching the extension, so the pub/sub half of §8 no longer storms.
- #2983 (open) introduces `PermanentConfigurationError` for the `upsertTransferConfig` task: permanent misconfigurations log and return instead of retrying; transient errors still retry.
- #2985 tracks the partitioning-field guard (§7b) separately.
## Possible remaining problem
`retry: false` may trade the storm for silent loss: a transient failure (Firestore blip, timeout) acks and drops the completion notification, and that run's output never reaches Firestore. The extension appears to have the same flaw, so this is parity, but arguably parity with a flaw. With `retry: true`, v2 pub/sub retries go through Eventarc with exponential backoff (10s to 600s, up to 24h), so retries look bounded rather than stormy - the storm concern seems to only apply to permanent errors, which the #2983 pattern addresses.
## Proposed plan (needs agreement, since it goes beyond parity)
1. Land #2963 (deterministic run-output ids) so redelivery cannot duplicate docs.
2. Extend the #2983 classification to `handleMessagePublished` (non-associated config, malformed payload → log and return).
3. Then flip `processMessages` to `retry: true`. Order matters.
4. Sweep the remaining handler throw sites and classify permanent vs transient.
The lost `setProcessingState` visibility is out of scope here, tracked in #3035.
Depends on: #2983, #2963, #2985. Parity ledger: #2974.
Contributor guide
Assessment
This issue has not been assessed yet.