apache / apache/beam

Support backfill deduplication in BigQueryIO.write()

Open
#18,414 0 comments 0 reactions 0 assignees View on GitHub
bug gcp io java P3
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
2d 2h
Merged PRs (30d)
205

Description

See https://github.com/GoogleCloudPlatform/DataflowJavaSDK/issues/603 motivated by SO question https://stackoverflow.com/questions/46076914/apache-beam-update-bigquery-table-row-with-bigqueryio

Perhaps one way we can do this is make BigQueryIO return a PValue that can be sequenced with other things, and implement a BigQuery.update() transform that executes a single DML statement (or a small collection thereof - since DML in BigQuery is very scarce), and let the user sandwich them together if they would like to.

Imported from Jira [BEAM-2864](https://issues.apache.org/jira/browse/BEAM-2864). Original Jira may contain additional context.
Reported by: jkff.

Contributor guide

Open the contributing guide

Research direction

Start with BigQueryIO.write() and read the linked DataflowJavaSDK issue 603, Stack Overflow question, and Jira BEAM-2864 for context. Define the desired backfill-deduplication behavior and assess the proposed sequenced PValue/BigQuery.update() approach; done means users can update BigQuery table rows through the supported transform without duplicate backfill results.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, sql
Domain
data, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.