Support backfill deduplication in BigQueryIO.write()
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 205
Description
See https://github.com/GoogleCloudPlatform/DataflowJavaSDK/issues/603 motivated by SO question https://stackoverflow.com/questions/46076914/apache-beam-update-bigquery-table-row-with-bigqueryio
Perhaps one way we can do this is make BigQueryIO return a PValue that can be sequenced with other things, and implement a BigQuery.update() transform that executes a single DML statement (or a small collection thereof - since DML in BigQuery is very scarce), and let the user sandwich them together if they would like to.
Imported from Jira [BEAM-2864](https://issues.apache.org/jira/browse/BEAM-2864). Original Jira may contain additional context.
Reported by: jkff.
Contributor guide
Research direction
Start with BigQueryIO.write() and read the linked DataflowJavaSDK issue 603, Stack Overflow question, and Jira BEAM-2864 for context. Define the desired backfill-deduplication behavior and assess the proposed sequenced PValue/BigQuery.update() approach; done means users can update BigQuery table rows through the supported transform without duplicate backfill results.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, sql
- Domain
- data, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100