fivetran / fivetran/dbt_unified_rag

[Feature] Further macro optimizations and potential refactoring of `dbt.concat` and `coalesce_cast`

Open
#21 0 comments 0 reactions 0 assignees View on GitHub
type:enhancement
Dominant language
No language data
Stars
6
Forks
7
PR merge metrics
No merged PRs in 30d

Description

### Is there an existing feature request for this?

- [x] I have searched the existing issues

### Describe the Feature

Several issues

1) `dbt.concat` does not handle the Snowflake null element case very well ([more details here](https://github.com/fivetran/dbt_unified_rag/issues/20) and could be rebuilt with our own macro to do so.

2) Currently the `coalesce_cast` macro is applied at the intermediate layer, which is non-standard--usually casting and coalescing would happen at the staging layer to reduce macro operations. This can lead to some of the base tables being `coalesce_cast`'ed at multiple intermediate models (like `user_name` and `created_by` in Zendesk) when it could just be cast in the staging layer once.

3) We do bring in a '1970-01-01 00:00:00.00' timestamp value for non-null cases in our coalesce_cast for timestamp values like `created_time`, which could lead to some confusion in the data if brought in. It'd be useful to figure a way to bring this value in as a similar UNKNOWN without specific

### How would you implement this feature?

Early thoughts:

1) we can create a custom `dbt.concat` macro that handles the snowflake null case we solved for.
2) We can explore bringing the macro upstream and apply it at the staging layer and see if it causes any issues.
3) Casting the timestamp value to string within `coalesce_cast` and bringing the original timestamp value downstream for any additional operations it's needed for.

### Describe alternatives you've considered

_No response_

### Are you interested in contributing this feature?

- [x] Yes.
- [ ] Yes, but I will need assistance.
- [ ] No.

### Anything else?

_No response_

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by locating the dbt.concat and coalesce_cast macros and the staging and intermediate models that use them, especially the Zendesk fields user_name, created_by, and created_time. Review the Snowflake null-element case and linked issue 20 first. Done should include an agreed approach for all three requested changes, with affected models checked for duplicate casting and timestamp behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
sql
Domain
data, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.