fivetran / fivetran/dbt_fivetran_utils

Macros to add/remove

Open
#158 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Shell
Stars
33
Forks
22
PR merge metrics
No merged PRs in 30d

Description

Starting to gather a housekeeping list of macros when we cut v0.5.0.

## Add
- Macros used in several packages
- Macros where we want to curtail divergence (i.e. packages have built their own local, near-duplicate versions instead of using the shared macro)
- Investigate if we can consolidate:
- `lookback`
- 8 packages (`dbt_mixpanel`, `dbt_amplitude`, `dbt_iterable`, `dbt_shopify`, `dbt_ga4_export`, `dbt_netsuite`, `dbt_jira`, `dbt_fivetran_log`) each carry an identical local `_lookback` macro. Confirmed byte-for-byte identical logic across all 8 (only differ by macro name and default `datepart`/`interval`/`safety_date` values).
- `extract_url_parameter`
- 10 packages (`dbt_facebook_ads`, `dbt_google_ads`, `dbt_linkedin`, `dbt_microsoft_ads`, `dbt_pinterest`, `dbt_reddit_ads`, `dbt_snapchat_ads`, `dbt_stackadapt`, `dbt_tiktok_ads`, `dbt_twitter`) have independently copy-pasted this macro under a package-prefixed name (e.g. `facebook_ads_extract_url_parameter`).

## Remove (move locally to packages)
- Macros used in <2 or fewer packages (count tbd) unless expected to increase.
- Macros where the `dbt_utils`/`dbt` equivalent is sufficient (mostly the cross-database compatibility macros below — `dbt-core` now ships its own cross-db macros, e.g. `dbt.try_cast`, which most packages call directly instead of ours).

### Used in 0 packages (Consider removal)
| Macro | Notes |
| -- | -- |
| add_dbt_source_relation | |
| ceiling | |
| collect_freshness | |
| drop_schemas_automation | |
| dummy_coalesce_value | |
| extract_url_parameter | see "Add" above - packages reimplemented locally instead of adopting; candidate to fix adoption rather than remove |
| first_value | cross-db compat - `dbt`/`dbt_utils` equivalent likely sufficient |
| json_extract | |
| seed_data_helper | obsolete CI/integration-test tooling only |
| snowflake_seed_data | obsolete CI/integration-test tooling only |
| wrap_in_quotes | |

### Used in exactly 1 package (Move locally to package)
| Macro | Package(s) |
| -- | -- |
| array_agg | dbt_hubspot |
| calculated_fields | dbt_hubspot |
| empty_variable_warning | dbt_zendesk |
| fivetran_date_spine | dbt_fivetran_log |
| remove_prefix_from_columns | dbt_hubspot |
| timestamp_add | dbt_zendesk |
| timestamp_diff | dbt_qualtrics |
| union_relations | dbt_marketo |

### Used in exactly 2 packages (Consider moving locally to package)
| Macro | Packages |
| -- | -- |
| enabled_vars_one_true | dbt_quickbooks, dbt_sage_intacct |
| try_cast | dbt_klaviyo, dbt_shopify_holistic_reporting |

### Reference - 3–5 packages, but probably worth keeping in fivetran_utils)
| Macro | Packages |
| -- | -- |
| max_bool | dbt_qualtrics, dbt_shopify, dbt_zendesk |
| percentile | dbt_intercom, dbt_jira, dbt_qualtrics |
| pivot_json_extract | dbt_amplitude, dbt_mixpanel, dbt_stripe |
| json_parse | dbt_fivetran_log, dbt_klaviyo, dbt_shopify, dbt_youtube_analytics, dbt_zendesk |

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by auditing the listed macros and their package usage, beginning with the duplicated lookback and extract_url_parameter implementations across the named packages. Review the zero-, one-, and two-package tables alongside the reference list, then document which macros should be shared, removed, or moved locally. Done means the v0.5.0 housekeeping list has confirmed decisions and no unresolved usage counts.

Written by the indexing model from the issue text.

Assessment

Domain
data-engineering
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.