aws-samples / aws-samples/dbt-glue

Support for Iceberg Partition Transforms

Open
#333 5 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
147
Forks
96
Avg merge
7h 4m
Merged PRs (30d)
5

Description

Hi guys,

It seems we are not able to use [Iceberg Partition Transforms](https://iceberg.apache.org/spec/#partition-transforms:~:text=Scan%20Planning%20below.-,Partition%20Transforms,-%F0%9F%94%97) inside dbt models.

**Table Config:**
{{ config(
materialized='incremental',
incremental_strategy='append',
file_format='iceberg',
**partition_by=['months(my_date)']**
) }}

**Output:**
CREATE TABLE glue_catalog.test.test_table
USING iceberg
PARTITIONED BY (months(my_date))
LOCATION 's3://sandbox-source/client/my/test/test_table'
AS SELECT * FROM tmp_test_table ORDER BY (months(my_date))

**Error: AnalysisException: Undefined function: months. This function is neither a built-in/temporary function, nor a persistent function that is qualified as spark_catalog.default.months.**

I guess ORDER BY is the root cause since iceberg doesn't support such kind of partition expressions in ORDER BY clause.
https://github.com/aws-samples/dbt-glue/blob/a52fcc49f93fcb3405a60a0cd627ca8d91726749/dbt/adapters/glue/impl.py#L874-L882

Contributor guide

Open the contributing guide

Research direction

Start in dbt/adapters/glue/impl.py around lines 874-882 and reproduce the issue using the Iceberg incremental model and partition_by example from the report. Inspect the generated CREATE TABLE statement and verify that partition transforms remain supported without producing an invalid ORDER BY expression.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python, sql
Domain
data-engineering, databases
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.