fivetran / fivetran/dbt_fivetran_log
Bug - fivetran_log__connector_status.sql failing with large sets of logs
- Dominant language
- No language data
- Stars
- 32
- Forks
- 30
- PR merge metrics
- No merged PRs in 30d
Description
### Is there an existing issue for this?
- [X] I have searched the existing issues
### Describe the issue
I have been running the fivetral_log dbt package on large sets of logs and I hit a memory limit (as per screenshot).

I have made some changes to my local version of fivetran_log__connector_status.sql model in order to get around the issue (see attached file) but could you investigate the issue?
[fivetran_log__connector_status.txt](https://github.com/fivetran/dbt_fivetran_log/files/8968486/fivetran_log__connector_status.txt)
### Relevant error log or model output
```shell
13:41:24 Database Error in model fivetran_log__connector_status (models/fivetran_log__connector_status.sql)
13:41:24 Resources exceeded during query execution: The query could not be executed in the allotted memory. Peak usage: 128% of limit.
13:41:24 Top memory consumer(s):
13:41:24 sort operations used for analytic OVER() clauses: 99%
13:41:24 other/unattributed: 1%
13:41:24
13:41:24 compiled SQL at target/run/fivetran_log/models/fivetran_log__connector_status.sql
```
### Expected behavior
Successful execution of the transformation
### dbt Project configurations
# Name your project! Project names should contain only lowercase characters
# and underscores. A good package name should reflect your organization's
# name or the intended use of these models
name: 'dbt_wildwest_fivetranlog'
version: '1.0.0'
config-version: 2
# This setting configures which "profile" dbt uses for this project.
profile: 'dbt_wildwest_fivetranlog'
# These configurations specify where dbt should look for different types of files.
# The `model-paths` config, for example, states that models in this project can be
# found in the "models/" directory. You probably won't need to change these!
model-paths: ["models"]
analysis-paths: ["analyses"]
test-paths: ["tests"]
seed-paths: ["seeds"]
macro-paths: ["macros"]
snapshot-paths: ["snapshots"]
target-path: "target" # directory which will store compiled SQL files
clean-targets: # directories to be removed by `dbt clean`
- "target"
- "dbt_packages"
# Configuring models
# Full documentation: https://docs.getdbt.com/docs/configuring-models
# In this example config, we tell dbt to build all models in the example/ directory
# as tables. These settings can be overridden in the individual model files
# using the `{{ config(...) }}` macro.
models:
fivetran_log:
# Config indicated by + and applies to all files under models/example/
# +post-hook: "delete FROM {{ var('connector') }} WHERE connector_name = 'julien_lambda_auth0';"
vars:
fivetran_log_database: fivetran-wild-west
fivetran_log_schema: julien_fivetran_log_live
# on-run-end:
# - SQL Statement
### Package versions
packages:
- package: fivetran/fivetran_log
version: 0.6.1
### What database are you using dbt with?
bigquery
### dbt Version
Core:
- installed: 1.1.0
Plugins:
- bigquery: 1.1.0
### Additional Context
_No response_
### Are you willing to open a PR to help address this issue?
- [X] Yes.
- [ ] Yes, but I will need assistance and will schedule time during our [office hours](https://calendly.com/fivetran-solutions-team/fivetran-solutions-team-office-hours) for guidance
- [ ] No.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.