Streaming BigQueryIO should support user-provided IDs
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 205
Description
Currently, BigQueryIO always assigns IDs and does a shuffle to ensure that they are atomic. This incurs a noticeable cost and is unnecessary if the user already has deterministic IDs that they can use. The sink should be able to use these IDs to skip the shuffle.
Imported from Jira [BEAM-384](https://issues.apache.org/jira/browse/BEAM-384). Original Jira may contain additional context.
Reported by: millsd@google.com.
Contributor guide
Research direction
Start by locating the Java BigQueryIO streaming sink and tracing its ID assignment and shuffle behavior; BEAM-384 may contain additional context. Done means deterministic user-provided IDs can be used by the sink and the unnecessary shuffle is skipped, with the relevant behavior covered by tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data, stream-processing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100