fivetran / fivetran/dbt_fivetran_log

Bug - fivetran_log__connector_status.sql failing with large sets of logs

Open
#56 3 comments 0 reactions 0 assignees View on GitHub
type:wontfix
Dominant language
No language data
Stars
32
Forks
30
PR merge metrics
No merged PRs in 30d

Description

### Is there an existing issue for this?

- [X] I have searched the existing issues

### Describe the issue

I have been running the fivetral_log dbt package on large sets of logs and I hit a memory limit (as per screenshot).

![image](https://user-images.githubusercontent.com/104826606/175331168-6b9b3a7b-c698-477b-bfac-212f3198e2dc.png)

I have made some changes to my local version of fivetran_log__connector_status.sql model in order to get around the issue (see attached file) but could you investigate the issue?

[fivetran_log__connector_status.txt](https://github.com/fivetran/dbt_fivetran_log/files/8968486/fivetran_log__connector_status.txt)

### Relevant error log or model output

```shell
13:41:24 Database Error in model fivetran_log__connector_status (models/fivetran_log__connector_status.sql)
13:41:24 Resources exceeded during query execution: The query could not be executed in the allotted memory. Peak usage: 128% of limit.
13:41:24 Top memory consumer(s):
13:41:24 sort operations used for analytic OVER() clauses: 99%
13:41:24 other/unattributed: 1%
13:41:24
13:41:24 compiled SQL at target/run/fivetran_log/models/fivetran_log__connector_status.sql
```

### Expected behavior

Successful execution of the transformation

### dbt Project configurations

# Name your project! Project names should contain only lowercase characters
# and underscores. A good package name should reflect your organization's
# name or the intended use of these models
name: 'dbt_wildwest_fivetranlog'
version: '1.0.0'
config-version: 2

# This setting configures which "profile" dbt uses for this project.
profile: 'dbt_wildwest_fivetranlog'

# These configurations specify where dbt should look for different types of files.
# The `model-paths` config, for example, states that models in this project can be
# found in the "models/" directory. You probably won't need to change these!
model-paths: ["models"]
analysis-paths: ["analyses"]
test-paths: ["tests"]
seed-paths: ["seeds"]
macro-paths: ["macros"]
snapshot-paths: ["snapshots"]

target-path: "target" # directory which will store compiled SQL files
clean-targets: # directories to be removed by `dbt clean`
- "target"
- "dbt_packages"

# Configuring models
# Full documentation: https://docs.getdbt.com/docs/configuring-models

# In this example config, we tell dbt to build all models in the example/ directory
# as tables. These settings can be overridden in the individual model files
# using the `{{ config(...) }}` macro.
models:
fivetran_log:
# Config indicated by + and applies to all files under models/example/
# +post-hook: "delete FROM {{ var('connector') }} WHERE connector_name = 'julien_lambda_auth0';"

vars:
fivetran_log_database: fivetran-wild-west
fivetran_log_schema: julien_fivetran_log_live

# on-run-end:
# - SQL Statement

### Package versions

packages:
- package: fivetran/fivetran_log
version: 0.6.1

### What database are you using dbt with?

bigquery

### dbt Version

Core:
- installed: 1.1.0
Plugins:
- bigquery: 1.1.0

### Additional Context

_No response_

### Are you willing to open a PR to help address this issue?

- [X] Yes.
- [ ] Yes, but I will need assistance and will schedule time during our [office hours](https://calendly.com/fivetran-solutions-team/fivetran-solutions-team-office-hours) for guidance
- [ ] No.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.