aws-samples / aws-samples/dbt-glue
Snapshot with check-col strategy and without datetime column throws an error
- Dominant language
- Python
- Stars
- 147
- Forks
- 96
- Avg merge
- 7h 4m
- Merged PRs (30d)
- 5
Description
### Describe the bug
When check strategy is chosen for snapshot without specifying a datetime column, dbt-glue calls the *datetime()* udf which is not available by default in Glue Interactive Session Spark SQL runtime.
### Steps To Reproduce
Run a dbt snapshot command with a snapshot config having: strategy='check', check_cols=['column_to_check']
Please find the below example for easily reproducibility :
- Copy and paste the below content into a csv file named student.csv.
```
id,firstname,lastname,DOB,age
1,Michael,P.,21/12/2007,15
2,Shawn,M.,12/06/1994,29
3,Kathleen,P.,13/01/1992,31
```
- Move the student.csv file in your dbt project seed path and run dbt seed --select student
- Create a snapshot with the below code. And run dbt snapshot --select demo_snap
```
{% snapshot demo_snap %}
{{
config(
target_schema='demo_db',
file_format='iceberg',
unique_key= " firstname || '-' || lastname",
strategy='check',
check_cols=['age'
,'DOB'
],
)
}}
SELECT
{{ cleanse_text_field('student', 'firstname ') }} AS firstname ,
{{ cleanse_text_field('student', 'lastname') }} AS lastname,
{{ cleanse_text_field('student', 'age') }} AS age,
TO_DATE({{ ref('student') }}.DOB , 'mm/dd/yyyy') AS DOB
FROM {{ ref('student') }}
{% endsnapshot %}
```
### Screenshots and log output
```
'datetime'. This function is neither a registered temporary function nor a permanent function registered in the database 'default'
```
### System information
**The output of `dbt --version`:**
```
- installed: 1.6.0
- latest: 1.6.0 - Up to date
Plugins:
- glue: 1.6.0 - Up to date!
- spark: 1.6.0 - Up to date!
```
Contributor guide
Research direction
Reproduce the failure with the supplied student seed and demo_snap configuration by running dbt seed --select student followed by dbt snapshot --select demo_snap. Start by tracing the check-strategy snapshot path and its use of the datetime() UDF in Glue Interactive Session Spark SQL. Done means a check-strategy snapshot without a datetime column completes without the unregistered-function error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python, spark
- Domain
- cloud, data
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100