Support BigQuery DML in BigQueryIO.Write
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 205
Description
Since 2016 it's possible to use DML functionality in BigQuery when using standard SQL. It would be great to also support this in Beam. Analogous to BigQueryIO.Read::fromQuery maybe a BigQueryIO.Write::withQuery method can be added, which is supported when standard SQL is enabled?
The reason why I think this is a major issue: within a few months GDPR goes into effect in the EU. This law enforces companies to support deleting all data from a certain customer. I think this feature would be useful in order to automate the deletion/fuzzing process of personal customer data.
Imported from Jira [BEAM-3313](https://issues.apache.org/jira/browse/BEAM-3313). Original Jira may contain additional context.
Reported by: wardva.
Contributor guide
Research direction
Start with the BigQueryIO.Write API and compare it with BigQueryIO.Read::fromQuery, then review how standard SQL is enabled for BigQuery operations. Done means a write-side withQuery method supports BigQuery DML when standard SQL is enabled, with behavior suitable for the stated deletion and data-fuzzing use case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, sql
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100