aws-samples / aws-samples/dbt-glue
Glue DB output override per SQL config
- Dominant language
- Python
- Stars
- 147
- Forks
- 96
- Avg merge
- 7h 4m
- Merged PRs (30d)
- 5
Description
I have been looking at overriding the output schema and database. Ideally we have 3 or 4 different glue databases. I have tried this within the following code, which allows me to override the S3 location but not the Glue DB:
{{ config(
partition_by=['year','month','day','hour'],
file_format='parquet',
materialized='table',
target_schema='bronze',
target_database='bronze',
custom_location=“s3://“ + env_var('AWS_ACCOUNT_ID') + “-bronze/data/general_ledger_posting_event",
tags=["gl"]
) }}
Ideally I would like to be able to override them at a higher level in the dbt_project.yml
models:
projectname:
# raw:
# +tags:
# - "raw"
# +materialized: table
bronze:
+file_format: hudi
+tags:
- "bronze"
+materialized: table
Contributor guide
Research direction
Start by tracing how target_schema, target_database, and custom_location are read from the model config and dbt_project.yml; the issue names no implementation files or tests. Compare the intended per-model and project-level overrides, then verify that separate Glue databases and schemas are honored with focused coverage.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python, sql
- Domain
- cloud, data-engineering, databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 32/100