dbt-labs / dbt-labs/dbt-adapters

[Regression] dbt-postgres: repeated dbt invocations in one process get slower over time since #1608

Open
#2,136 0 comments 0 reactions 0 assignees View on GitHub
triage:product type:regression
Dominant language
Python
Stars
233
Forks
362
Avg merge
3d 22h
Merged PRs (30d)
9

Description

## Is this a regression?

- [x] I believe this is a regression in functionality
- [x] I have searched the existing issues, and I could not find an existing issue for this regression

## Which packages are affected?

- [x] dbt-postgres

## Current Behavior

When a single Python process runs dbt more than once, each run costs more than the one before it. The work is identical every time, but the time per run keeps climbing.

Running one functional test 60 times in a single pytest process, on identical code apart from the adapter:

| dbt-postgres | iterations 1-20 | 21-40 | 41-60 | growth |
| --- | --- | --- | --- | --- |
| `8c2272169` (parent of #1608) | 1.499s | 1.469s | 1.442s | 0.96x |
| `1683729be` (#1608) | 1.741s | 2.158s | 2.646s | 1.52x |
| 1.11.0 (released) | 1.748s | 2.185s | 2.623s | 1.50x |
| 1.10.2 (released) | 1.522s | 1.489s | 1.454s | 0.96x |

The released 1.11.0 behaves the same as the commit, so this is not limited to git main. The 1.10.x line never received #1608 and stays flat.

Database work is not the cause. Every arm above issued exactly 125 queries and opened 42 connections per iteration, and time inside `SQLConnectionManager.add_query` stayed flat at roughly 1.05s while total time went from 1.7s to 2.6s. All of the growth lands outside the query, connection, parse, and macro paths.

Memory is not the cause either. Live object count held at about 400k, GC time held at 0.086s to 0.093s per iteration, and worker RSS sat between 198MB and 301MB after 54 minutes and 1400 tests.

I tried to narrow it further by neutralising parts of #1608 at runtime on 1.11.0:

| intervention | mean per iteration | growth |
| --- | --- | --- |
| none (1.11.0 as shipped) | 2.090s | 1.37x |
| remove `begin` / `commit` / `rollback_if_open` overrides | 1.995s | 1.42x |
| remove `PostgresAdapter.__init__` override | 1.983s | 1.39x |
| make `_behavior_flags` return `[]` | 1.602s | 1.30x |
| pre-#1608 parent commit | 1.484s | 0.96x |

Dropping the `_behavior_flags` declaration recovers the constant factor. Starting time goes back to 1.531s, against 1.534s for the parent commit. The growth survives all four interventions, so I have not found the line responsible for it.

Two things that surprised me and may help whoever picks this up. On the default path the new code is never reached: across five full iterations, `_should_skip_transaction_statements`, `_is_autocommit_enabled`, the checker closure, and the `behavior` property were each called zero times, because `autocommit` defaults to `False` and short circuits first. Yet removing the `_behavior_flags` declaration still changes the timing, so something reads `_behavior_flags` directly rather than going through `behavior`.

Setting `autocommit: true` does what it promises and does not help here. Query count drops from 125 to 95 and database time falls 21%, but growth is unchanged at 1.38x, and the mean stays 28% worse than pre-#1608.

## Expected/Previous Behavior

Per invocation cost stays flat for the life of the process, as it does on 1.10.2 and on the parent commit of #1608.

## Steps To Reproduce

```bash
pip install "dbt-core~=1.12.0" "dbt-postgres==1.11.0" pytest pytest-repeat
git clone https://github.com/dbt-labs/dbt-core && cd dbt-core && git checkout 1.12.latest
pip install -e core

# postgres on localhost, roles per scripts/setup_db.sh
python -m pytest --count=60 -q \
tests/functional/basic/test_simple_reference.py::test_simple_reference
```

Compare against `dbt-postgres==1.10.2`. Time the first and last ten iterations rather than the total, since the total hides the shape.

Any repeated in process invocation shows it. I used dbt-core's functional suite because it was the workload where I hit this.

## Relevant log output

Per iteration instrumentation on 1.11.0, wrapping `add_query`, `PostgresConnectionManager.open`, `ManifestLoader.load`, and `execute_macro`:

```
iters total db dbq conn connN parse macro macroN other
1-30 1.837 1.066 125 0.377 42.0 0.210 0.498 56 0.184
31-60 2.563 1.161 125 0.424 42.0 0.204 0.552 56 0.775
61-90 3.276 1.145 125 0.430 42.0 0.205 0.567 56 1.496
91-120 3.958 1.145 125 0.436 42.0 0.203 0.562 56 2.175
121-150 4.684 1.150 125 0.427 42.0 0.204 0.558 56 2.903
```

`db`, `conn`, `parse`, and `macro` are flat. `other` accounts for all of the growth.

## Environment

```
- OS: macOS 15 (arm64), also reproduced on ubuntu-24.04 in CI
- Python: 3.10.19
- dbt-core: 1.12.3
- dbt-adapters: 1.24.5
- dbt-common: 1.39.0
- dbt-postgres: 1.11.0 (and 1.11.0rc1, and 1683729be)
- PostgreSQL server: 18.6
```

## Additional Context

This is costing real release time. dbt-core's release pipeline runs the functional suite unsharded in one job. On the 1.12 line, which installs dbt-postgres from git main, that job takes about 2h45m. On the 1.11 line, which installs dbt-postgres from PyPI at 1.10.2, the same suite takes about 10m38s.

The nightly release job dates the onset precisely. It ran at roughly 430s per night through 2026-02-10, then jumped to roughly 5500s on 2026-02-11 and has stayed there, drifting up to roughly 9900s by August. #1608 merged at 2026-02-11T00:56Z, between the last fast nightly and the first slow one, and it is the only adapters commit in that window.

PR based CI never caught this because it shards the suite into 15 groups of about 100 tests each, one fresh process per group. The effect needs several hundred invocations in one process to become obvious.

Users running dbt through a long lived process are the ones exposed: anything driving `dbtRunner` repeatedly, such as dbt Cloud, the Python bindings, or an MCP server. I have not tested whether a single large `dbt run` degrades as it walks thousands of nodes, which would matter much more. `begin` and `commit` fire per node, so the same mechanism could apply, but I have no measurement for it.

There is no way to turn this off from configuration. `autocommit` already defaults to `False`, and the behaviour flag `postgres_skip_autocommit_transaction_statements` is never consulted on that path. Pinning `dbt-postgres<1.11` is the only mitigation I found, which also means giving up the autocommit feature that #1608 added.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.