aws-samples / aws-samples/dbt-glue

Unable to create new model in iceberg format from existing model which is in iceberg format

Open
#140 2 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
147
Forks
96
Avg merge
7h 4m
Merged PRs (30d)
5

Description

### Describe the bug
I have created one model `sample` with following configuration and dbt created glue catalog table

```
{{
config(
materialized="incremental",
file_format='iceberg',
incremental_strategy='insert_overwrite',
unique_key=['dt'],
table_properties={'write.target-file-size-bytes': '268435456'}
)
}}

WITH base AS (
SELECT SEQUENCE(
DATE '2014-01-01',
DATE_ADD(current_date(),1),
interval '1' day
) AS seq
),
dates AS(
SELECT EXPLODE(seq) AS dt
FROM base
)

SELECT
*
FROM dates
```

Now when I create another model just selecting from above model with following configuration

```
{{
config(
materialized="incremental",
file_format='iceberg',
incremental_strategy='insert_overwrite',
unique_key=['dt'],
table_properties={'write.target-file-size-bytes': '268435456'}
)
}}

select * from {{ref('sample') }}
```

I am getting following error

```
11:16:59 Glue adapter: Glue returned `error` for statement None for code SqlWrapper2.execute('''/* {"app": "dbt", "dbt_version": "1.3.2", "profile_name": "default", "target_name": "dev", "node_id": "model.dbt_eda.sample1"} */
select * from glue_catalog.smullangi.sample1 limit 1 '''), AnalysisException: Table or view not found: glue_catalog.smullangi.sample1; line 2 pos 14;
'GlobalLimit 1
+- 'LocalLimit 1
+- 'Project [*]
+- 'UnresolvedRelation [glue_catalog, smullangi, sample1], [], false

11:16:59 1 of 1 ERROR creating sql incremental model smullangi.sample1 .................. [ERROR in 35.18s]
11:17:01
11:17:01 Finished running 1 incremental model in 0 hours 1 minutes and 19.13 seconds (79.13s).
11:17:02
11:17:02 Completed with 1 error and 0 warnings:
11:17:02
11:17:02 Database Error in model sample1 (models/utils/sample1.sql)
11:17:02 Glue cursor returned `error` for statement None for code SqlWrapper2.execute('''/* {"app": "dbt", "dbt_version": "1.3.2", "profile_name": "default", "target_name": "dev", "node_id": "model.dbt_eda.sample1"} */
11:17:02 select * from glue_catalog.smullangi.sample1 limit 1 '''), AnalysisException: Table or view not found: glue_catalog.smullangi.sample1; line 2 pos 14;
11:17:02 'GlobalLimit 1
11:17:02 +- 'LocalLimit 1
11:17:02 +- 'Project [*]
11:17:02 +- 'UnresolvedRelation [glue_catalog, smullangi, sample1], [], false
11:17:02
11:17:02 compiled Code at target/run/dbt_eda/models/utils/sample1.sql
11:17:02
11:17:02 Done. PASS=0 WARN=0 ERROR=1 SKIP=0 TOTAL=1
```

### Steps To Reproduce
1. create a file name sample.sql with configuration mentioned above
2. Perform dbt run --model sample
3. Create another file sample1.sql with configuration mentioned above
4. perform dbt run --model sample1
5. Now we can see error as mentioned

### Expected behavior
dbt should create catalog table for sample1 in iceberg format

### Screenshots and log output
If applicable, add screenshots or log output to help explain your problem.

### System information
**The output of `dbt --version`:**
```
Core:
- installed: 1.3.2
- latest: 1.4.1 - Update available!

Your version of dbt-core is out of date!
You can find instructions for upgrading here:
https://docs.getdbt.com/docs/installation

Plugins:
- glue: 1.3.12 - Up to date!
- spark: 1.3.1 - Update available!

At least one plugin is out of date or incompatible with dbt-core.
You can find instructions for upgrading here:
https://docs.getdbt.com/docs/installation
```

**The operating system you're using:** Mac

**The output of `python --version`:**
```Python 3.9.7```

### Additional context
This is happening only when we create tables in iceberg format. If we dont mention file_format, by default it picks parquet and its running correctly.

Contributor guide

Open the contributing guide

Research direction

First reproduce the two dbt runs with the reported dbt 1.3.2 and Glue adapter 1.3.12 versions, comparing the Iceberg and default Parquet configurations. Inspect target/run/dbt_eda/models/utils/sample1.sql and the reported Glue error; done means the sample1 catalog table is created successfully in Iceberg format.

Written by the indexing model from the issue text.

Assessment

Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.