GoogleCloudPlatform / GoogleCloudPlatform/firebase-extensions

[bigquery-firestore-export]: Scalability Issues

Open
#863 1 comment 0 reactions 0 assignees View on GitHub
extension: bigquery-firestore-export reproduced type: bug
Dominant language
TypeScript
Stars
124
Forks
67
Avg merge
5d 5h
Merged PRs (30d)
5

Description

This extension suffers from scalability issues:

It currently tries to pull all rows from the BQ query into memory, and write them to firestore. This causes OOM and timeout for large queries.

This could be improved by moving to a v2 function (allows for greater memory and longer timeout). We would need to double check all triggers are supported by the function.

A more long-term solution would be to fan-out and push chunks of rows to a Cloud Task queue, though this would take more design work to ensure it's robust and deterministic.

This was the real reason behind https://github.com/firebase/extensions/issues/2544

Expected behaviour:

The extension is able to process queries with millions of rows

Actual behaviour:

The function times out or runs out of memory.

TODO:

- [ ] upgrade to v2
- [ ] redesign extension to fan out

(we will track these as sub-issues as they are prioritised)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.