fivetran / fivetran/dbt_pinterest
[Bug] Constant Expressions Error for Redshift
- Dominant language
- No language data
- Stars
- 3
- Forks
- 10
- PR merge metrics
- No merged PRs in 30d
Description
### Is there an existing issue for this?
- [X] I have searched the existing issues
### Describe the issue
#### Issue Details
Since the introduction of the union data feature in [v0.10.0](https://github.com/fivetran/dbt_pinterest_source/releases/tag/v0.10.0) we have seen an issue be introduced where Redshift users who are not leveraging the union data feature are seeing a constant expression error (see relevant error log for full error output). This error is caused because the union data macro will create the `source_relation` field and will insert an empty string if no union schemas/databases are provided in the respective variables. Therefore, since we have the empty string for all fields it is treated as a constant expression and therefore is not able to be included in a partition statement.
This issue commonly only happens when the upstream source table is not present (the package creates an empty staging model), but it can also happen when the upstream source table is present. We previously addressed this issue by applying the [following code update](https://github.com/fivetran/dbt_facebook_ads_source/blob/bb99b05240fb4a96064d2b83099d9ea20423a196/models/stg_facebook_ads__account_history.sql#L40-L42) in other Ad Reporting packages.
```sql
case when id is null and _fivetran_synced is null
then row_number() over (partition by source_relation order by source_relation)
else row_number() over (partition by source_relation, id order by _fivetran_synced desc) end = 1 as is_most_recent_record
```
While this does seem to work to address this constant expression issue when the upstream table is not present, it has been found to not address the issue when the constant expression issue is persisting even when the upstream source table is present.
Thankfully, in the recent release of the [dbt_aws_cloud_cost](https://github.com/fivetran/dbt_aws_cloud_cost) dbt package we found a creative solution to this issue! Instead of the lengthy case when statement from before, we found that we can remove the `source_relation` field from the partition if the respective union schemas/databases variable is empty (since this issue does not occur when using the union schema variable. Therefore, the [following code update](https://github.com/fivetran/dbt_aws_cloud_cost/blob/c36385e570a31c211ce1523fbff82a7aeb799e06/models/aws_cloud_cost__daily_overview.sql#L76) in dbt_aws_cloud_cost addressed this very issue.
```sql
row_number() over (partition by bill_payer_account_id {{ ", source_relation" if var('aws_cloud_cost_sources', []) | length > 1 }} order by latest_start_date desc) = 1 as is_latest_name
```
#### Solution
We found that in some cases one solution works but the other doesn't. Fortunately, in the latest [Twitter Organic PR fivetran/dbt_pinterest_source#11](https://github.com/fivetran/dbt_twitter_organic/pull/11) we found a solution to this very problem. To completely address the constant expression issue we will do the following:
- Create a macro titled [is_table_emtpy](https://github.com/fivetran/dbt_twitter_organic/blob/update/default-schema-change/macros/is_table_empty.sql) and change the dispatch to reference `pinterest_source`.
- Creata a macro titled [result_if_table_exists](https://github.com/fivetran/dbt_twitter_organic/blob/update/default-schema-change/macros/result_if_table_exists.sql) and change the dispatch to reference `pinterest_source`.
- Where we do the window functions we apply a similar update to [this](https://github.com/fivetran/dbt_twitter_organic/blob/dfd3639a06fc03e912038d6d3ffb0728f0cd830c/models/intermediate/int_twitter_organic__latest_account.sql#L10-L14).
- Apply the above macro reference to each impacted window function in this package and any downstream cases.
This will then need to be applied to all other window functions in this package. Please ensure all window functions receive this update. From my initial scope I imagine this only needs to be applied to the `is_most_recent_record` fields in the `*_history` staging models.
### Relevant error log or model output
```shell
constant expressions are not supported in partition by clauses compiled code
```
### Expected behavior
The models are able to compile successfully on Redshift regardless of the constant expressions generated from the union data feature.
### dbt Project configurations
Normal schema and database variables. No other custom configuration.
### Package versions
```yml
packages:
- package: fivetran/pinterest_source
version: [">=0.10.0", "<0.11.0"]
```
### What database are you using dbt with?
redshift
### dbt Version
v1.8.x
### Additional Context
It may also be worthwhile to inspect the downstream models to ensure we do or do not need to make similar updates there.
### Are you willing to open a PR to help address this issue?
- [ ] Yes.
- [ ] Yes, but I will need assistance and will schedule time during our [office hours](https://calendly.com/fivetran-solutions-team/fivetran-solutions-team-office-hours) for guidance
- [ ] No.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.