dbt-labs / dbt-labs/dbt-adapters

[Bug] Large Time Gaps Between dbt Statements (Suspected Python GIL Issue)

Open
#786 6 comments 0 reactions 0 assignees View on GitHub
feature:transactions pkg:dbt-redshift type:bug
Dominant language
Python
Stars
233
Forks
362
Avg merge
3d 22h
Merged PRs (30d)
9

Description

### Is this a new bug in dbt-redshift?

- [x] I believe this is a new bug in dbt-redshift
- [x] I have searched the existing issues, and I could not find an existing issue for this bug

### Current Behavior

**Description:**

We're observing significant time gaps (up to 2 minutes) between dbt statements (COMMIT and BEGIN/ROLLBACK) in our Redshift data pipeline, leading to increased model processing times. This issue affects multiple models and occurs in various stages of the dbt run. Initial investigation points to a potential bottleneck related to the Python Global Interpreter Lock (GIL).

**Steps to Reproduce:**

1. Run a dbt project with multiple models (e.g., 2000+ models).
2. Observe the dbt logs and Redshift query history.
3. Notice large time gaps (minutes) between COMMIT and subsequent BEGIN/ROLLBACK statements for individual models.

**Example:**

https://getdbt.slack.com/archives/CJARVS0RY/p1738171778055219

**Observed Behavior:**

* With multiple threads (e.g., 5), we see significant time gaps between dbt statements, resulting in longer overall processing times (e.g., 3.5 hours for ~2k models).
* With a single thread, the time gaps disappear, and the processing time increases (e.g., 4.5 hours for ~2k models), but the individual model execution is more consistent. (See attached screenshot of single-threaded execution without time gaps).
* Processing with 5 threads is not even 2x faster, let alone 5x faster, than single-threaded execution.

**Suspected Root Cause:**

The observed behavior strongly suggests a bottleneck related to the Python GIL. When multiple threads are used, the GIL likely prevents true parallel execution of dbt operations, leading to queuing and delays between statements.

**Workaround:**

Running dbt with a single thread eliminates the time gaps, but significantly increases the overall processing time.

**Impact:**

Increased dbt run times, impacting data freshness and downstream processes.

**Attachments:**

* Screenshots from Slack thread showing dbt logs and Redshift query history with time gaps.
* Screenshot of single-threaded execution without time gaps.

**Questions:**

* Can the dbt-redshift adapter be optimized to minimize GIL contention?
* Are there any recommended configurations or best practices for running dbt with multiple threads on Redshift to avoid this issue?
* if we have autocommit in dbt profile it should not submit any transactional statements at all
* if there are no hooks it should not submit 2 fake commands BEGIN/ROLLBACK
* Can dbt take benefit from the new option(GIL becomes optional) introduced in python 3.13? (edited)
(https://gtm-roche.slack.com/archives/D03G68BDEDS/p1738743200437589)
https://medium.com/@mitesh.singh.jat/gil-becomes-optional-in-python-3-13-a-game-changer-for-multithreading-4c5d28856803

### Expected Behavior

dbt statements within a single model should execute consecutively with minimal delay.

### Steps To Reproduce

* dbt-redshift adapter version: 1.8.1
* dbt version: 1.8.9
* Redshift cluster: (Please specify cluster type and size if possible)
* MWAA (Managed Workflows for Apache Airflow) - dbt runs are orchestrated through MWAA.
* autocommit: enabled

### Relevant log output

```shell

```

### Environment

```markdown
- OS:
- Python:
- dbt-core:
- dbt-redshift:
- node type: ra3.16xlarge
- autoWLM: No, reverted to the manual tuned WLM
```

### Additional Context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.