fivetran / fivetran/dbt_netsuite
[Bug] Performance degradation by clustering/sorting
- Dominant language
- No language data
- Stars
- 57
- Forks
- 42
- Avg merge
- 6d 9h
- Merged PRs (30d)
- 2
Description
### Is there an existing issue for this?
- [X] I have searched the existing issues
### Describe the issue
we have observed much longer runtime with "cluster_by = ['transaction_id']," introduced in v0.13.0 release; the code refence is here: https://github.com/fivetran/dbt_netsuite/pull/117/files#diff-144d46b313d4b1851f6b2a20d16c25a6f41758af76cb008d874c1f61530383f3R7 and is with 3 major models.
seems the sorting takes about 40% of runtime and snowflake warns about the high cardinatlity of transcation_id is causing long runtime.
we have tried locally to compare; on balance_sheet model - 25 minutes with current code, clustering enables vs 14 minutes with clustering removed.
### Relevant error log or model output
_No response_
### Expected behavior
shorter execution time
### dbt Project configurations
dbt-core==1.7.14
dbt-snowflake==1.7.5
### Package versions
- package: dbt-labs/dbt_utils
version: 1.1.1
- package: brooklyn-data/dbt_artifacts
version: 2.3.0
- package: dbt-labs/dbt_project_evaluator
version: 0.8.0
- package: dbt-labs/dbt_external_tables
version: 0.8.7
### What database are you using dbt with?
snowflake
### dbt Version
dbt-core==1.7.14
dbt-snowflake==1.7.5
### Additional Context
_No response_
### Are you willing to open a PR to help address this issue?
- [ ] Yes.
- [X] Yes, but I will need assistance and will schedule time during our [office hours](https://calendly.com/fivetran-solutions-team/fivetran-solutions-team-office-hours) for guidance
- [ ] No.
Contributor guide
No contributing guide indexed for this repository
Research direction
Review the clustering change from dbt_netsuite PR #117 and inspect the three affected major models, including balance_sheet. Compare execution times with clustering enabled and removed, then determine whether the clustering configuration should change so the affected models complete faster without losing required behavior.
Written by the indexing model from the issue text.
Assessment
- Domain
- databases, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100