googleapis / googleapis/google-cloud-node
Add deadline capability to bulk writer
- Dominant language
- TypeScript
- Stars
- 3.2k
- Forks
- 712
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 99
Description
**Is your feature request related to a problem? Please describe.**
We use both `BulkWriter` and `recursiveDelete` in our Cloud Functions, which have a limit to how long they can run an execution for. It is rare that we need longer than the allowed run time to perform the writes, but it can happen on occasion and it would be useful to have a graceful mechanism to chunk the writes.
**Describe the solution you'd like**
There are two solutions I have thought of, and I'm open to other ideas of course.
1. Add a deadline option to bulk writers, either in the form of epoch milliseconds or as the number of milliseconds the bulk writer is allowed to run (i.e., 480000 for 8 minutes). If the expiration time is reached before the bulk writer has finished, then it stops making write requests and drops any queued writes not yet sent. The bulk writer would then reject with a deadline exceeded type of error we can handle.
2. Add a function to bulk writer that we can call to cancel any queued writes. Similar to the `flush` function except that it would drop any pending writes instead of trying to commit them. We could then call this ourselves after we ourselves determine that the deadline has passed.
Either option would allow us to then repeat the process to "chunk" the writes across multiple Cloud Functions executions when necessary.
**Describe alternatives you've considered**
We can either let the function time itself out, killing the bulk writer, or we can put a limit on the number of writes allowed by guesstimating how long it would take to complete. Both options are...less than ideal. In my opinion, the best solution would be a combination of limiting the BulkWriter time and limiting the total number of writes we feed the bulk writer, such that we don't end up reading exponentially more documents than written (as in many cases we need to read first, in order to determine what to write).
This issue is not critical because it's not very often that we need more time than the environment allows, but it would be a useful feature regardless.
Contributor guide
Assessment
This issue has not been assessed yet.