dbt-labs / dbt-labs/dbt-project-evaluator

Fusion: find_all_hard_coded_references fails with regex backtracking limit on large raw-SQL model

Open
#595 3 comments 0 reactions 0 assignees View on GitHub
bug triage
Dominant language
Shell
Stars
577
Forks
93
Avg merge
9h 46m
Merged PRs (30d)
1

Description

### Describe the bug

`dbt_project_evaluator` v1.3.4 fails on dbt Fusion when `stg_nodes` evaluates a large raw-SQL model. The failure occurs in the package's `default__find_all_hard_coded_references` macro, at the `re.findall(...)` call that scans each model's full raw SQL against several broad regex patterns.

### Steps to reproduce

1. Have a large raw-SQL model in the project — in our case, `mart_merchant_transaction_features`, approximately 2,762 lines / 74 KB, containing a wide generated feature `SELECT` and a normal `FROM {{ ref(...) }}` CTE reference.
2. Run `dbt build --select stg_nodes` (or any invocation that triggers `dbt_project_evaluator`'s graph scan) under Fusion.
3. The failure occurs while `stg_nodes` iterates through the graph and calculates `hard_coded_references` for every SQL model — it reaches this specific model and the regex scan throws the error below.

Note that the package's exception/exclusion mechanism does not help avoid this failure, because exceptions are applied only after `stg_nodes` has already evaluated the regex.

### Expected results

The evaluator should complete the graph scan, or skip/report the specific model without failing the entire evaluator job.

### Actual results

The whole `dbt build` invocation fails; no evaluator output is produced for any model, not just the large one.

### Screenshots and log output

```text
Error executing regex: Max limit for backtracking count exceeded
```

Stack trace points to:

```text
macros/unpack/find_all_hard_coded_references.sql:242
re.findall(regex_pattern, model_raw_sql_no_comments)
```

### System information

**The contents of your `packages.yml` file:**

```yaml
packages:
- package: dbt-labs/dbt_project_evaluator
version: 1.3.4
```

**Which database are you using dbt with?**
- [ ] postgres
- [ ] redshift
- [x] bigquery
- [ ] snowflake
- [ ] trino/starburst
- [ ] other (specify: ____________)

**The output of `dbt --version`:**
```
dbt-fusion 2.0.0-preview.212
```

### Additional context

The regex patterns in `find_all_hard_coded_references` run over each model's entire raw SQL. On this model, they trigger Fusion's regex engine backtracking limit. Related but distinct: #581 reports a different Fusion incompatibility in this same macro (`match[1:]` failing with "value of type map cannot be sliced").

### Are you interested in contributing the fix?

No.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.