aws-samples / aws-samples/dbt-glue

Snapshot with check-col strategy and without datetime column throws an error

Open
#217 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
147
Forks
96
Avg merge
7h 4m
Merged PRs (30d)
5

Description

### Describe the bug
When check strategy is chosen for snapshot without specifying a datetime column, dbt-glue calls the *datetime()* udf which is not available by default in Glue Interactive Session Spark SQL runtime.

### Steps To Reproduce
Run a dbt snapshot command with a snapshot config having: strategy='check', check_cols=['column_to_check']
Please find the below example for easily reproducibility :
- Copy and paste the below content into a csv file named student.csv.
```
id,firstname,lastname,DOB,age
1,Michael,P.,21/12/2007,15
2,Shawn,M.,12/06/1994,29
3,Kathleen,P.,13/01/1992,31
```
- Move the student.csv file in your dbt project seed path and run dbt seed --select student
- Create a snapshot with the below code. And run dbt snapshot --select demo_snap
```
{% snapshot demo_snap %}
{{
config(
target_schema='demo_db',
file_format='iceberg',
unique_key= " firstname || '-' || lastname",
strategy='check',
check_cols=['age'
,'DOB'
],
)
}}
SELECT
{{ cleanse_text_field('student', 'firstname ') }} AS firstname ,
{{ cleanse_text_field('student', 'lastname') }} AS lastname,
{{ cleanse_text_field('student', 'age') }} AS age,
TO_DATE({{ ref('student') }}.DOB , 'mm/dd/yyyy') AS DOB
FROM {{ ref('student') }}

{% endsnapshot %}
```

### Screenshots and log output
```
'datetime'. This function is neither a registered temporary function nor a permanent function registered in the database 'default'
```
### System information
**The output of `dbt --version`:**
```
- installed: 1.6.0
- latest: 1.6.0 - Up to date
Plugins:
- glue: 1.6.0 - Up to date!
- spark: 1.6.0 - Up to date!
```

Contributor guide

Open the contributing guide

Research direction

Reproduce the failure with the supplied student seed and demo_snap configuration by running dbt seed --select student followed by dbt snapshot --select demo_snap. Start by tracing the check-strategy snapshot path and its use of the datetime() UDF in Glue Interactive Session Spark SQL. Done means a check-strategy snapshot without a datetime column completes without the unregistered-function error.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python, spark
Domain
cloud, data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.