firebase / firebase/extensions
[bigquery-firestore-export kit] Retried runs re-add every row, multiplying output documents
- Dominant language
- TypeScript
- Stars
- 979
- Forks
- 433
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 125
Description
In the `bigquery-firestore-export` kit (`@firebase/bigquery-firestore-export`, on the `kits` branch), a run that fails partway through writing its rows duplicates its whole output on every retry.
### Cause
Rows are written with `collection.add()` in `kits/bigquery-firestore-export/src/helper.ts`, which mints a new document id per call, and the run document under `.../runs/` is only written once every row has been written. A run that dies partway leaves its rows in place, and the retry writes the whole result set again under fresh ids.
`processMessages` sets `retry: true` (`kits/bigquery-firestore-export/src/index.ts`), so the notification is redelivered until the subscription's retention expires, 24 hours by default. Nothing dedupes and nothing cleans up the partial attempt.
### What happens
With a 3,000 row source on a live project, watched for 14 minutes: 3,400, then 6,400, then 9,848, then 12,900 documents under `.../runs//output`, still climbing when the subscription was purged.
### Fix
Write rows with deterministic ids derived from the run and the row index, so a retry overwrites rather than appends.
Contributor guide
Research direction
Start in kits/bigquery-firestore-export/src/helper.ts, where rows are written with collection.add(), and read kits/bigquery-firestore-export/src/index.ts to understand the retry path. Change row writes to use deterministic IDs derived from the run and row index, then verify that a retry overwrites existing output documents instead of adding duplicates.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- firebase, typescript
- Domain
- database
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 82/100