Consider making insertId optional in BigQuery.insertAll
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
Current implementation of StreamingWriteFn(https://github.com/apache/beam/blob/c2f0d282337f3ae0196a7717712396a5a41fdde1/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/StreamingWriteFn.java#L102) sets insertId from input element, which is added an uniqueId by https://github.com/apache/beam/blob/c2f0d282337f3ae0196a7717712396a5a41fdde1/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/TagWithUniqueIds.java#L53. Users report that if leaving insertId as empty, writing will be extremely speeded up. Can we add an bqOption like, nonInsertId and emit empty id based on this option?
Imported from Jira [BEAM-8222](https://issues.apache.org/jira/browse/BEAM-8222). Original Jira may contain additional context.
Reported by: boyuanz.
Contributor guide
Research direction
Start with the cited StreamingWriteFn.java and TagWithUniqueIds.java locations, and trace how insertId is assigned before BigQuery insertAll calls. Review the existing BigQuery IO tests and verify that the new option can select empty IDs while the current behavior remains unchanged.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- google-cloud, java
- Domain
- cloud, data-engineering
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100