aws-samples / aws-samples/dbt-glue
Support for Iceberg Partition Transforms
- Dominant language
- Python
- Stars
- 147
- Forks
- 96
- Avg merge
- 7h 4m
- Merged PRs (30d)
- 5
Description
Hi guys,
It seems we are not able to use [Iceberg Partition Transforms](https://iceberg.apache.org/spec/#partition-transforms:~:text=Scan%20Planning%20below.-,Partition%20Transforms,-%F0%9F%94%97) inside dbt models.
**Table Config:**
{{ config(
materialized='incremental',
incremental_strategy='append',
file_format='iceberg',
**partition_by=['months(my_date)']**
) }}
**Output:**
CREATE TABLE glue_catalog.test.test_table
USING iceberg
PARTITIONED BY (months(my_date))
LOCATION 's3://sandbox-source/client/my/test/test_table'
AS SELECT * FROM tmp_test_table ORDER BY (months(my_date))
**Error: AnalysisException: Undefined function: months. This function is neither a built-in/temporary function, nor a persistent function that is qualified as spark_catalog.default.months.**
I guess ORDER BY is the root cause since iceberg doesn't support such kind of partition expressions in ORDER BY clause.
https://github.com/aws-samples/dbt-glue/blob/a52fcc49f93fcb3405a60a0cd627ca8d91726749/dbt/adapters/glue/impl.py#L874-L882
Contributor guide
Research direction
Start in dbt/adapters/glue/impl.py around lines 874-882 and reproduce the issue using the Iceberg incremental model and partition_by example from the report. Inspect the generated CREATE TABLE statement and verify that partition transforms remain supported without producing an invalid ORDER BY expression.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python, sql
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100