apache / apache/incubator-xtable
Create BigQuery Sink
- Dominant language
- Java
- Stars
- 1.2k
- Forks
- 212
- Avg merge
- 4d 9h
- Merged PRs (30d)
- 16
Description
Build a BigQuery sync will allow customers to incrementally sync from table formats to BigLake Managed Tables (and BQ native tables).
The implementation of the sink will be different than other sinks:
1. Instead of metadata translation, it will invoke a `LOAD` statement in BigQuery.
2. It can only support added files and not removed files.
@the-other-tim-brown: Do you think this will be useful?
Contributor guide
No contributing guide indexed for this repository
Research direction
No files, tests, or entry points are named. Start by examining the existing sinks referenced in the issue and clarify the BigLake Managed Tables and BigQuery native-table targets, along with added-file-only behavior. Done means an incremental BigQuery sink that invokes LOAD rather than metadata translation and does not handle removed files.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- google-cloud, java
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100