aws-samples / aws-samples/dbt-glue
Unable to create new model in iceberg format from existing model which is in iceberg format
- Dominant language
- Python
- Stars
- 147
- Forks
- 96
- Avg merge
- 7h 4m
- Merged PRs (30d)
- 5
Description
### Describe the bug
I have created one model `sample` with following configuration and dbt created glue catalog table
```
{{
config(
materialized="incremental",
file_format='iceberg',
incremental_strategy='insert_overwrite',
unique_key=['dt'],
table_properties={'write.target-file-size-bytes': '268435456'}
)
}}
WITH base AS (
SELECT SEQUENCE(
DATE '2014-01-01',
DATE_ADD(current_date(),1),
interval '1' day
) AS seq
),
dates AS(
SELECT EXPLODE(seq) AS dt
FROM base
)
SELECT
*
FROM dates
```
Now when I create another model just selecting from above model with following configuration
```
{{
config(
materialized="incremental",
file_format='iceberg',
incremental_strategy='insert_overwrite',
unique_key=['dt'],
table_properties={'write.target-file-size-bytes': '268435456'}
)
}}
select * from {{ref('sample') }}
```
I am getting following error
```
11:16:59 Glue adapter: Glue returned `error` for statement None for code SqlWrapper2.execute('''/* {"app": "dbt", "dbt_version": "1.3.2", "profile_name": "default", "target_name": "dev", "node_id": "model.dbt_eda.sample1"} */
select * from glue_catalog.smullangi.sample1 limit 1 '''), AnalysisException: Table or view not found: glue_catalog.smullangi.sample1; line 2 pos 14;
'GlobalLimit 1
+- 'LocalLimit 1
+- 'Project [*]
+- 'UnresolvedRelation [glue_catalog, smullangi, sample1], [], false
11:16:59 1 of 1 ERROR creating sql incremental model smullangi.sample1 .................. [ERROR in 35.18s]
11:17:01
11:17:01 Finished running 1 incremental model in 0 hours 1 minutes and 19.13 seconds (79.13s).
11:17:02
11:17:02 Completed with 1 error and 0 warnings:
11:17:02
11:17:02 Database Error in model sample1 (models/utils/sample1.sql)
11:17:02 Glue cursor returned `error` for statement None for code SqlWrapper2.execute('''/* {"app": "dbt", "dbt_version": "1.3.2", "profile_name": "default", "target_name": "dev", "node_id": "model.dbt_eda.sample1"} */
11:17:02 select * from glue_catalog.smullangi.sample1 limit 1 '''), AnalysisException: Table or view not found: glue_catalog.smullangi.sample1; line 2 pos 14;
11:17:02 'GlobalLimit 1
11:17:02 +- 'LocalLimit 1
11:17:02 +- 'Project [*]
11:17:02 +- 'UnresolvedRelation [glue_catalog, smullangi, sample1], [], false
11:17:02
11:17:02 compiled Code at target/run/dbt_eda/models/utils/sample1.sql
11:17:02
11:17:02 Done. PASS=0 WARN=0 ERROR=1 SKIP=0 TOTAL=1
```
### Steps To Reproduce
1. create a file name sample.sql with configuration mentioned above
2. Perform dbt run --model sample
3. Create another file sample1.sql with configuration mentioned above
4. perform dbt run --model sample1
5. Now we can see error as mentioned
### Expected behavior
dbt should create catalog table for sample1 in iceberg format
### Screenshots and log output
If applicable, add screenshots or log output to help explain your problem.
### System information
**The output of `dbt --version`:**
```
Core:
- installed: 1.3.2
- latest: 1.4.1 - Update available!
Your version of dbt-core is out of date!
You can find instructions for upgrading here:
https://docs.getdbt.com/docs/installation
Plugins:
- glue: 1.3.12 - Up to date!
- spark: 1.3.1 - Update available!
At least one plugin is out of date or incompatible with dbt-core.
You can find instructions for upgrading here:
https://docs.getdbt.com/docs/installation
```
**The operating system you're using:** Mac
**The output of `python --version`:**
```Python 3.9.7```
### Additional context
This is happening only when we create tables in iceberg format. If we dont mention file_format, by default it picks parquet and its running correctly.
Contributor guide
Research direction
First reproduce the two dbt runs with the reported dbt 1.3.2 and Glue adapter 1.3.12 versions, comparing the Iceberg and default Parquet configurations. Inspect target/run/dbt_eda/models/utils/sample1.sql and the reported Glue error; done means the sample1 catalog table is created successfully in Iceberg format.
Written by the indexing model from the issue text.
Assessment
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100