redpanda-data / redpanda-data/connect
New output component for BigQuery for the new Write API
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 8.8k
- Forks
- 969
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 64
Description
As the current BigQuery component is still experimental, it would be good to replace it with a new component using the Write API from Big Query: BigQuery Write API explained: An overview of the Write API
The advantage of using this api is that it uses Protobuf data. Which means it's faster as it will be moving binary data around.
Here are some additional material: Learn how to stream JSON data into BigQuery using the new BigQuery Storage Write API
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the current BigQuery component linked in the issue, then review the referenced BigQuery Storage Write API material and the repository’s existing output-component conventions. Done means a new BigQuery output component using the Write API is implemented and its behavior is verified consistently with the project’s existing components.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, google-cloud
- Domain
- data-engineering, stream-processing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100