fivetran / fivetran/dbt_netsuite

[Bug] Performance degradation by clustering/sorting

Open
#127 5 comments 0 reactions 0 assignees View on GitHub
error:unforced status:stale type:bug
Dominant language
No language data
Stars
57
Forks
42
Avg merge
6d 9h
Merged PRs (30d)
2

Description

### Is there an existing issue for this?

- [X] I have searched the existing issues

### Describe the issue

we have observed much longer runtime with "cluster_by = ['transaction_id']," introduced in v0.13.0 release; the code refence is here: https://github.com/fivetran/dbt_netsuite/pull/117/files#diff-144d46b313d4b1851f6b2a20d16c25a6f41758af76cb008d874c1f61530383f3R7 and is with 3 major models.

seems the sorting takes about 40% of runtime and snowflake warns about the high cardinatlity of transcation_id is causing long runtime.

we have tried locally to compare; on balance_sheet model - 25 minutes with current code, clustering enables vs 14 minutes with clustering removed.

### Relevant error log or model output

_No response_

### Expected behavior

shorter execution time

### dbt Project configurations

dbt-core==1.7.14
dbt-snowflake==1.7.5

### Package versions

- package: dbt-labs/dbt_utils
version: 1.1.1
- package: brooklyn-data/dbt_artifacts
version: 2.3.0
- package: dbt-labs/dbt_project_evaluator
version: 0.8.0
- package: dbt-labs/dbt_external_tables
version: 0.8.7

### What database are you using dbt with?

snowflake

### dbt Version

dbt-core==1.7.14
dbt-snowflake==1.7.5

### Additional Context

_No response_

### Are you willing to open a PR to help address this issue?

- [ ] Yes.
- [X] Yes, but I will need assistance and will schedule time during our [office hours](https://calendly.com/fivetran-solutions-team/fivetran-solutions-team-office-hours) for guidance
- [ ] No.

Contributor guide

No contributing guide indexed for this repository

Research direction

Review the clustering change from dbt_netsuite PR #117 and inspect the three affected major models, including balance_sheet. Compare execution times with clustering enabled and removed, then determine whether the clustering configuration should change so the affected models complete faster without losing required behavior.

Written by the indexing model from the issue text.

Assessment

Domain
databases, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.