fivetran / fivetran/dbt_unified_rag
[Feature] Further macro optimizations and potential refactoring of `dbt.concat` and `coalesce_cast`
- Dominant language
- No language data
- Stars
- 6
- Forks
- 7
- PR merge metrics
- No merged PRs in 30d
Description
### Is there an existing feature request for this?
- [x] I have searched the existing issues
### Describe the Feature
Several issues
1) `dbt.concat` does not handle the Snowflake null element case very well ([more details here](https://github.com/fivetran/dbt_unified_rag/issues/20) and could be rebuilt with our own macro to do so.
2) Currently the `coalesce_cast` macro is applied at the intermediate layer, which is non-standard--usually casting and coalescing would happen at the staging layer to reduce macro operations. This can lead to some of the base tables being `coalesce_cast`'ed at multiple intermediate models (like `user_name` and `created_by` in Zendesk) when it could just be cast in the staging layer once.
3) We do bring in a '1970-01-01 00:00:00.00' timestamp value for non-null cases in our coalesce_cast for timestamp values like `created_time`, which could lead to some confusion in the data if brought in. It'd be useful to figure a way to bring this value in as a similar UNKNOWN without specific
### How would you implement this feature?
Early thoughts:
1) we can create a custom `dbt.concat` macro that handles the snowflake null case we solved for.
2) We can explore bringing the macro upstream and apply it at the staging layer and see if it causes any issues.
3) Casting the timestamp value to string within `coalesce_cast` and bringing the original timestamp value downstream for any additional operations it's needed for.
### Describe alternatives you've considered
_No response_
### Are you interested in contributing this feature?
- [x] Yes.
- [ ] Yes, but I will need assistance.
- [ ] No.
### Anything else?
_No response_
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by locating the dbt.concat and coalesce_cast macros and the staging and intermediate models that use them, especially the Zendesk fields user_name, created_by, and created_time. Review the Snowflake null-element case and linked issue 20 first. Done should include an agreed approach for all three requested changes, with affected models checked for duplicate casting and timestamp behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- sql
- Domain
- data, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100