Handle BigQuery loading failures in Cloud Functions
- Dominant language
- Python
- Stars
- 26
- Forks
- 40
- PR merge metrics
- No merged PRs in 30d
Description
BigQuery loading can fail sometimes. E.g. due to insufficient schema validation described in #108 data could be accepted into the queue, but rejected by BigQuery. In this case it could be beneficial to log the problem and acknowledge the broken pub/sub message, so it's removed from the queue. Otherwise it will keep getting picked up by the Cloud Functions and keep failing forever.
One way to do this could be by dividing the pulled list of messages into two, in case of such a failure, and retrying the loading until the culprit(s) are found, logged and ACKed, and the rest of the messages is loaded and ACKed. This should observe the load job flow control. I.e. dividing should only be done until the first load job succeeds, and the rest of retrying should be left to the queue mechanisms.
Contributor guide
No contributing guide indexed for this repository
Research direction
No file or test is named. Start by locating the Cloud Functions BigQuery loading entry point and its Pub/Sub acknowledgement flow; verify how load-job flow control is handled. Done means failed messages are logged and acknowledged, remaining messages are loaded and acknowledged, and retries do not continue forever.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- google-cloud, python
- Domain
- cloud, data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100