aws-samples / aws-samples/dbt-glue

Glue DB output override per SQL config

Open
#68 2 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
147
Forks
96
Avg merge
7h 4m
Merged PRs (30d)
5

Description

I have been looking at overriding the output schema and database. Ideally we have 3 or 4 different glue databases. I have tried this within the following code, which allows me to override the S3 location but not the Glue DB:

{{ config(
partition_by=['year','month','day','hour'],
file_format='parquet',
materialized='table',
target_schema='bronze',
target_database='bronze',
custom_location=“s3://“ + env_var('AWS_ACCOUNT_ID') + “-bronze/data/general_ledger_posting_event",
tags=["gl"]
) }}

Ideally I would like to be able to override them at a higher level in the dbt_project.yml
models:
projectname:
# raw:
# +tags:
# - "raw"
# +materialized: table
bronze:
+file_format: hudi
+tags:
- "bronze"
+materialized: table

Contributor guide

Open the contributing guide

Research direction

Start by tracing how target_schema, target_database, and custom_location are read from the model config and dbt_project.yml; the issue names no implementation files or tests. Compare the intended per-model and project-level overrides, then verify that separate Glue databases and schemas are honored with focused coverage.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python, sql
Domain
cloud, data-engineering, databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.