BigQueryIO.write() with DynamicDestinations should not call getSchema() on every element
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 2d 5h
- Merged PRs (30d)
- 204
Description
https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/CreateTables.java#L103 calls getSchema() on every element. This might be very expensive if the user expects that this will be called about once per destination.
Imported from Jira [BEAM-3083](https://issues.apache.org/jira/browse/BEAM-3083). Original Jira may contain additional context.
Reported by: jkff.
Contributor guide
Research direction
Start in sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/CreateTables.java around line 103 and trace how BigQueryIO.write() with DynamicDestinations obtains schemas. Verify the current getSchema() call frequency, then confirm completion when schema lookup is no longer performed for every element and occurs about once per destination.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100