fivetran / fivetran/dbt_apple_store

[Bug] Queries taking too long

Open
#23 25 comments 0 reactions 0 assignees View on GitHub
error:forced status:accepted type:bug
Dominant language
No language data
Stars
1
Forks
3
Avg merge
39m
Merged PRs (30d)
1

Description

### Is there an existing issue for this?

- [X] I have searched the existing issues

### Describe the issue

Around about the 2nd August I upgraded (all packages and dbt) but in particular this package from 0.3.0 to 0.4.0

Since then the load on my small PostgreSQL instance has gone from averaging at 11% to 90%

![image](https://github.com/user-attachments/assets/5dc72768-c3a8-4e9d-b31e-8fffc1315fa6)

What I see is the query below running for hours

The only tables in this query with significant rows are the territory tables but they are circa 124000 rows only.

I'm running PostgreSQL in Azure. I'm no DB expert so haven't been really able to dig into this.

Happy to debug and help as needed or be told I'm doing it wrong :)

### Relevant error log or model output

```shell
CREATE TABLE
"dwh_prod"."dbt_apple_store"."apple_store__territory_report__dbt_tmp"
AS (
WITH
app AS (SELECT * FROM "dwh_prod"."dbt_apple_store_stg"."stg_apple_store__app"),
app_store_territory AS (SELECT * FROM "dwh_prod"."dbt_apple_store_stg"."stg_apple_store__app_store_territory"),
country_codes AS (SELECT * FROM "dwh_prod"."dbt_apple_store_source"."apple_store_country_codes"),
downloads_territory AS (SELECT * FROM "dwh_prod"."dbt_apple_store_stg"."stg_apple_store__downloads_territory"),
usage_territory AS (SELECT * FROM "dwh_prod"."dbt_apple_store_stg"."stg_apple_store__usage_territory"),
reporting_grain AS (SELECT DISTINCT source_relation, date_day, app_id, source_type, territory FROM app_store_territory),
joined AS (
SELECT
reporting_grain.source_relation,
reporting_grain.date_day,
reporting_grain.app_id,
app.app_name,
reporting_grain.source_type,
reporting_grain.territory AS territory_long,
coalesce(official_country_codes.country_code_alpha_2, alternative_country_codes.country_code_alpha_2) AS territory_short,
coalesce(official_country_codes.region, alternative_country_codes.region) AS region,
coalesce(official_country_codes.sub_region, alternative_country_codes.sub_region) AS sub_region,
coalesce(app_store_territory.impressions, 0) AS impressions,
coalesce(app_store_territory.impressions_unique_device, 0) AS impressions_unique_device,
coalesce(app_store_territory.page_views, 0) AS page_views,
coalesce(app_store_territory.page_views_unique_device, 0) AS page_views_unique_device,
coalesce(downloads_territory.first_time_downloads, 0) AS first_time_downloads,
coalesce(downloads_territory.redownloads, 0) AS redownloads,
coalesce(downloads_territory.total_downloads, 0) AS total_downloads,
coalesce(usage_territory.active_devices, 0) AS active_devices,
coalesce(usage_territory.active_devices_last_30_days, 0) AS active_devices_last_30_days,
coalesce(usage_territory.deletions, 0) AS deletions,
coalesce(usage_territory.installations, 0) AS installations,
coalesce(usage_territory.sessions, 0) AS sessions
FROM reporting_grain
LEFT JOIN app
on reporting_grain.app_id = app.app_id
AND reporting_grain.source_relation = app.source_relation
LEFT JOIN app_store_territory
on reporting_grain.date_day = app_store_territory.date_day
AND reporting_grain.source_relation = app_store_territory.source_relation
AND reporting_grain.app_id = app_store_territory.app_id
AND reporting_grain.source_type = app_store_territory.source_type
AND reporting_grain.territory = app_store_territory.territory
LEFT JOIN downloads_territory
on reporting_grain.date_day = downloads_territory.date_day
AND reporting_grain.source_relation = downloads_territory.source_relation
AND reporting_grain.app_id = downloads_territory.app_id
AND reporting_grain.source_type = downloads_territory.source_type
AND reporting_grain.territory = downloads_territory.territory
LEFT JOIN usage_territory
on reporting_grain.date_day = usage_territory.date_day
AND reporting_grain.source_relation = usage_territory.source_relation
AND reporting_grain.app_id = usage_territory.app_id
AND reporting_grain.source_type = usage_territory.source_type
AND reporting_grain.territory = usage_territory.territory
LEFT JOIN country_codes AS official_country_codes
on reporting_grain.territory = official_country_codes.country_name
LEFT JOIN country_codes AS alternative_country_codes
on reporting_grain.territory = alternative_country_codes.alternative_country_name
)
SELECT * FROM joined;
```

### Expected behavior

Query completes within a reasonable time

### dbt Project configurations

``` yml
name: 'gladly'
version: '1.0.0'
config-version: 2

profile: 'gladly'

model-paths: ["models"]
analysis-paths: ["analyses"]
test-paths: ["tests"]
seed-paths: ["seeds"]
macro-paths: ["macros"]
snapshot-paths: ["snapshots"]

target-path: "target"
clean-targets:
- "target"
- "dbt_packages"

vars:

# Stripe settings
stripe_timezone: "Australia/Sydney"
stripe_schema: stripe_load
stripe__using_invoices: False
stripe__using_subscriptions: False
stripe__using_credit_notes: False
stripe__using_subscription_history: True

# Play Store
google_play_schema: google_play_load

# App Store
apple_store_schema: itunes_connect_load

# Amplitude
amplitude_schema: amplitude_load

# Xero
xero_schema: xero_load

# Twitter Ads
twitter_ads_schema: twitter_ads_load
twitter_ads__using_keywords: False # Our account doesn't use keywords

# Facebook Ads
facebook_ads_schema: facebook_ads_load

# Ad Reporting
ad_reporting__apple_search_ads_enabled: False
ad_reporting__pinterest_ads_enabled: False
ad_reporting__microsoft_ads_enabled: False
ad_reporting__linkedin_ads_enabled: False
ad_reporting__google_ads_enabled: False
ad_reporting__snapchat_ads_enabled: False
ad_reporting__tiktok_ads_enabled: False
ad_reporting__amazon_ads_enabled: False
ad_reporting__reddit_ads_enabled: False

# Linkedin pages
linkedin_pages_schema: linkedin_company_pages_load

# Twitter organic
twitter_organic_schema: twitter_organic_load

# Youtube analytics
youtube_analytics_schema: youtube_analytics_load
youtube__using_video_metadata: true
youtube_metadata_schema: youtube_analytics_load
youtube__channel_basic_table: channel_basic_a_2
youtube__channel_demographics_table: channel_demographics_a_1

# Facebook Pages
facebook_pages_schema: facebook_ads_load

# Instagram Business
instragram_business_schema: instragram_business_load

# Dynamo DB Settings
dynamodb_schema: dynamodb_load
dynamodb_suffix: qry_4_ki_4_hvzhm_5_bnv_26_qtcqscaq_prod

models:
stripe_source:
+schema: stripe_stg

google_play_source:
+schema: google_play_stg

apple_store_source:
+schema: apple_store_stg

amplitude_source:
+schema: amplitude_stg

xero_source:
+schema: xero_stg

twitter_ads_source:
+schema: twitter_ads_stg

twitter_organic_source:
+schema: twitter_organic_stg

facebook_ads_source:
+schema: facebook_ads_stg

linkedin_pages_source:
+schema: linkedin_company_pages_stg

youtube_analytics_source:
+schema: youtube_analytics_stg

facebook_pages_source:
+schema: facebook_pages_stg

instagram_business_source:
+schema: instagram_business_stg

gladly:
dynamodb:
+schema: gladly
+materialized: view
stats:
+schema: stats
+materialized: view
```

### Package versions

```yml
packages:
- package: fivetran/stripe
version: [">=0.14.0", "<0.15.0"]

- package: fivetran/google_play
version: [">=0.4.0", "<0.5.0"]

- package: fivetran/apple_store
version: [">=0.4.0", "<0.5.0"]

- package: fivetran/app_reporting
version: [">=0.4.0", "<0.5.0"]

- package: fivetran/amplitude
version: [">=0.4.0", "<0.5.0"]

- package: fivetran/xero
version: [">=0.6.0", "<0.7.0"]

- package: fivetran/twitter_ads
version: [">=0.7.0", "<0.8.0"]

- package: fivetran/facebook_ads
version: [">=0.7.0", "<0.8.0"]

- package: fivetran/ad_reporting
version: [">=1.9.0", "<1.10.0"]

- package: fivetran/linkedin_pages
version: [">=0.3.0", "<0.4.0"]

- package: fivetran/twitter_organic
version: [">=0.2.0", "<0.3.0"]

- package: fivetran/youtube_analytics
version: [">=0.4.0", "<0.5.0"]

- package: fivetran/facebook_pages
version: [">=0.3.0", "<0.4.0"]

- package: fivetran/instagram_business
version: [">=0.2.0", "<0.3.0"]

- package: fivetran/social_media_reporting
version: [">=0.4.0", "<0.5.0"]
```

### What database are you using dbt with?

postgres

### dbt Version

1.7.17 (Note I'm also using dbt transformations, same issue in both places)

### Additional Context

_No response_

### Are you willing to open a PR to help address this issue?

- [X] Yes.
- [ ] Yes, but I will need assistance and will schedule time during our [office hours](https://calendly.com/fivetran-solutions-team/fivetran-solutions-team-office-hours) for guidance
- [ ] No.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the long-running CREATE TABLE query included in the issue and compare its behavior between package versions 0.3.0 and 0.4.0 on PostgreSQL. The issue names no repository files or tests; done means identifying the regression and making the affected query complete within a reasonable time, with a reproducible check.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql
Domain
databases, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.