apache / apache/beam

Support BigQuery DML in BigQueryIO.Write

Open
#18,528 0 comments 0 reactions 0 assignees View on GitHub
gcp io java new feature P3
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
2d 2h
Merged PRs (30d)
205

Description

Since 2016 it's possible to use DML functionality in BigQuery when using standard SQL. It would be great to also support this in Beam. Analogous to BigQueryIO.Read::fromQuery maybe a BigQueryIO.Write::withQuery method can be added, which is supported when standard SQL is enabled?

The reason why I think this is a major issue: within a few months GDPR goes into effect in the EU. This law enforces companies to support deleting all data from a certain customer. I think this feature would be useful in order to automate the deletion/fuzzing process of personal customer data.

Imported from Jira [BEAM-3313](https://issues.apache.org/jira/browse/BEAM-3313). Original Jira may contain additional context.
Reported by: wardva.

Contributor guide

Open the contributing guide

Research direction

Start with the BigQueryIO.Write API and compare it with BigQueryIO.Read::fromQuery, then review how standard SQL is enabled for BigQuery operations. Done means a write-side withQuery method supports BigQuery DML when standard SQL is enabled, with behavior suitable for the stated deletion and data-fuzzing use case.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, sql
Domain
databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.