aws-samples / aws-samples/dbt-glue
Enable Model-Level Target Catalog Configuration for S3Tables
- Dominant language
- Python
- Stars
- 147
- Forks
- 96
- Avg merge
- 7h 4m
- Merged PRs (30d)
- 5
Description
### Describe the feature
Add functionality enabling the specification of target catalog in the model config blocks (or for groups of models via dbt_project.yml file) similarly to how custom schemas work. Currently it is possible to override the default target catalog identifier: "glue_catalog" using the _custom_iceberg_catalog_namespace_ parameter in the profiles.yml file, but this still only allows for specifying a single target catalog for each DBT run. We would like the ability to perform writes to multiple distinct catalogs/s3 table buckets in a single DBT run execution.
### Describe alternatives you've considered
Our current job setup uses two separate tasks that each perform individual DBT runs to write to each target catalog/s3 table bucket. However, we would like to combine them to benefit from DBT's query sequencing and dependency management capabilities.
### Additional context
We have our spark config set to recognize multiple catalogs:
```
--conf spark.sql.catalog.glue_catalog=org.apache.iceberg.spark.SparkCatalog
--conf spark.sql.defaultCatalog=glue_catalog
--conf spark.sql.catalog.glue_catalog.warehouse=s3://{{ env_var('DBT_ENV_NAME') }}-dbt-glue-warehouse-{{ env_var('DBT_AWS_ACCOUNT_ID') }}
--conf spark.sql.catalog.glue_catalog.catalog-impl=org.apache.iceberg.aws.glue.GlueCatalog
--conf spark.sql.catalog.glue_catalog.io-impl=org.apache.iceberg.aws.s3.S3FileIO
--conf spark.sql.catalog.glue_catalog.glue.id={{ env_var('DBT_AWS_ACCOUNT_ID') }}:s3tablescatalog/{{ env_var('DBT_ENV_NAME') }}-{{ env_var('DBT_S3TABLES_TARGET_BUCKET') }}
--conf spark.sql.catalog.spark_catalog=org.apache.iceberg.spark.SparkSessionCatalog
--conf spark.sql.catalog.spark_catalog.catalog-impl=org.apache.iceberg.aws.glue.GlueCatalog
--conf spark.sql.catalog.bronze=org.apache.iceberg.spark.SparkCatalog
--conf spark.sql.catalog.bronze.warehouse=s3://{{ env_var('DBT_ENV_NAME') }}-dbt-glue-warehouse-{{ env_var('DBT_AWS_ACCOUNT_ID') }}
--conf spark.sql.catalog.bronze.catalog-impl=org.apache.iceberg.aws.glue.GlueCatalog
--conf spark.sql.catalog.bronze.io-impl=org.apache.iceberg.aws.s3.S3FileIO
--conf spark.sql.catalog.bronze.glue.id={{ env_var('DBT_AWS_ACCOUNT_ID') }}:s3tablescatalog/{{ env_var('DBT_ENV_NAME') }}-s3tables-bronze
--conf spark.sql.catalog.silver=org.apache.iceberg.spark.SparkCatalog
--conf spark.sql.catalog.silver.warehouse=s3://{{ env_var('DBT_ENV_NAME') }}-dbt-glue-warehouse-{{ env_var('DBT_AWS_ACCOUNT_ID') }}
--conf spark.sql.catalog.silver.catalog-impl=org.apache.iceberg.aws.glue.GlueCatalog
--conf spark.sql.catalog.silver.io-impl=org.apache.iceberg.aws.s3.S3FileIO
--conf spark.sql.catalog.silver.glue.id={{ env_var('DBT_AWS_ACCOUNT_ID') }}:s3tablescatalog/{{ env_var('DBT_ENV_NAME') }}-s3tables-silver
--conf spark.sql.catalog.gold=org.apache.iceberg.spark.SparkCatalog
--conf spark.sql.catalog.gold.warehouse=s3://{{ env_var('DBT_ENV_NAME', 'dev') }}-deng-lakehouse-otp-dbt-glue-warehouse-{{ env_var('DBT_AWS_ACCOUNT_ID') }}
--conf spark.sql.catalog.gold.catalog-impl=org.apache.iceberg.aws.glue.GlueCatalog
--conf spark.sql.catalog.gold.io-impl=org.apache.iceberg.aws.s3.S3FileIO
--conf spark.sql.catalog.gold.glue.id={{ env_var('DBT_AWS_ACCOUNT_ID') }}:s3tablescatalog/{{ env_var('DBT_ENV_NAME') }}-s3tables-gold
```
This allows us to successfully read from tables in multiple catalogs within a single run by listing their aliases explicitly in our queries, but we have no method to achieve similar capabilities with writes.
### Who will this benefit?
Anyone using the adapter to write to multiple catalogs / s3 table buckets
### Are you interested in contributing this feature?
Yes
Contributor guide
Research direction
Start by tracing how custom_iceberg_catalog_namespace is handled from profiles.yml, then compare that path with model config blocks and dbt_project.yml configuration. The work is done when models or model groups can select distinct configured catalogs and a single dbt run writes successfully to multiple S3 Tables buckets.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python, spark
- Domain
- cloud, data-engineering, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100