[destination-bigquery] Optionally materialize output tables as BigQuery-managed Apache Iceberg tables (table_format=ICEBERG)
- Linguagem predominante
- Python
- Estrelas
- 22.1k
- Forks
- 5.3k
- Métricas de merge de PRs
- Métricas de PR pendentes
Descrição
# [destination-bigquery] Optionally materialize output tables as BigQuery-managed Apache Iceberg tables (table_format=ICEBERG)
## Problem
`destination-bigquery` (currently v3.0.21) writes standard **native** BigQuery tables via batched standard inserts / GCS staging + load jobs. There is no option to have it produce **BigQuery-managed Apache Iceberg tables** ("BigQuery tables for Apache Iceberg"), where data lives in a customer-owned GCS bucket in open Parquet/Iceberg format while BigQuery remains the metadata owner and sole writer.
Today the only Iceberg-capable GCS destination is `destination-gcs-data-lake`, but that is an **Iceberg REST-catalog client** (BigLake/Polaris): it writes snapshots to GCS and commits them via the REST catalog. Tables written that way surface in BigQuery as **read-only, catalog-federated external tables** — no BigQuery DML, no fine-grained access control, and they require the user to own Iceberg compaction/maintenance. It also maps `Number → Iceberg Double` (FLOAT64), which is **lossy for monetary/decimal data**.
## Proposed feature
Add an optional table-format setting to `destination-bigquery` that, when enabled, creates its destination tables as **BigQuery-managed Iceberg tables**, i.e. the connector issues `CREATE TABLE ... WITH CONNECTION OPTIONS(file_format='PARQUET', table_format='ICEBERG', storage_uri='gs:///...')` and continues to load/mutate via BigQuery (Storage Write API / load / DML), so BigQuery stays the writer/owner.
New config (opt-in, default off — preserves current native-table behaviour):
- `table_format`: `NATIVE` (default) | `ICEBERG`
- `biglake_connection` / cloud-resource connection id (required when `ICEBERG`)
- `storage_uri` / GCS bucket prefix for table data (required when `ICEBERG`)
## Why it matters
Because BigQuery is still the writer, the output is a genuine **managed** Iceberg table, which uniquely combines:
- **Open format + multi-engine access** — data in customer-owned GCS as Iceberg V2/Parquet, readable by Spark/Trino/other engines, while remaining a first-class BigQuery table.
- **Governed & DML-capable** — full GoogleSQL DML, column-level security, and data masking (unlike the read-only federated tables the REST-catalog path yields).
- **Auto-compaction / zero maintenance** — BigQuery does adaptive file sizing, automatic clustering, garbage collection, and metadata optimization, removing the customer-owned Iceberg maintenance burden that the lake connector imposes.
- **Precision preserved** — `destination-bigquery` already maps `NUMBER → NUMERIC`, so monetary/decimal data keeps precision (vs `Number → Double`/FLOAT64 in gcs-data-lake).
## Alternatives considered
- **dbt-bigquery managed-Iceberg materialization (GA in dbt-bigquery 1.10)** — configure a `biglake_metastore` catalog (`external_volume`, `table_format: iceberg`, `file_format: parquet`) and materialize models as managed Iceberg tables. This works today with no Airbyte change and is the right approach when a transformation layer already exists. It does not cover the "land raw Airbyte output directly as managed Iceberg" case.
- **`destination-gcs-data-lake`** — produces read-only, catalog-federated external Iceberg tables (no BQ DML, no FGAC, customer-owned compaction, lossy decimals). It does **not**, and by design **cannot**, produce managed tables: it commits via the Iceberg REST catalog, so BigQuery is not the writer. Filing this request against `destination-gcs-data-lake` would be incorrect — a REST-catalog writer inherently yields the federated/external object, which is the categorical opposite of a managed table. `destination-bigquery` is the only Airbyte connector where BigQuery is the writer and this capability is architecturally coherent.
## References
- BigQuery tables for Apache Iceberg (managed): https://cloud.google.com/bigquery/docs/iceberg-tables
- BigLake metastore Iceberg REST catalog (read-only/federated from BQ): https://cloud.google.com/bigquery/docs/blms-rest-catalog
- dbt-bigquery managed Iceberg support: https://docs.getdbt.com/docs/mesh/iceberg/bigquery-iceberg-support
---
**Internal Tracking:** https://github.com/airbytehq/oncall/issues/13025
Guia de contribuição
Avaliação
Esta issue ainda não foi avaliada.