firebase / firebase/extensions

[bigquery-firestore-export kit] Retried runs re-add every row, multiplying output documents

Open Beginner friendly
#2,971 0 comments 0 reactions 0 assignees View on GitHub
type: bug
Dominant language
TypeScript
Stars
979
Forks
433
Avg merge
1d 13h
Merged PRs (30d)
125

Description

In the `bigquery-firestore-export` kit (`@firebase/bigquery-firestore-export`, on the `kits` branch), a run that fails partway through writing its rows duplicates its whole output on every retry.

### Cause

Rows are written with `collection.add()` in `kits/bigquery-firestore-export/src/helper.ts`, which mints a new document id per call, and the run document under `.../runs/` is only written once every row has been written. A run that dies partway leaves its rows in place, and the retry writes the whole result set again under fresh ids.

`processMessages` sets `retry: true` (`kits/bigquery-firestore-export/src/index.ts`), so the notification is redelivered until the subscription's retention expires, 24 hours by default. Nothing dedupes and nothing cleans up the partial attempt.

### What happens

With a 3,000 row source on a live project, watched for 14 minutes: 3,400, then 6,400, then 9,848, then 12,900 documents under `.../runs//output`, still climbing when the subscription was purged.

### Fix

Write rows with deterministic ids derived from the run and the row index, so a retry overwrites rather than appends.

Contributor guide

Open the contributing guide

Research direction

Start in kits/bigquery-firestore-export/src/helper.ts, where rows are written with collection.add(), and read kits/bigquery-firestore-export/src/index.ts to understand the retry path. Change row writes to use deterministic IDs derived from the run and row index, then verify that a retry overwrites existing output documents instead of adding duplicates.

Written by the indexing model from the issue text.

Assessment

Tech stack
firebase, typescript
Domain
database
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
82/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.