dbt-labs / dbt-labs/dbt-project-evaluator
Fusion: find_all_hard_coded_references fails with regex backtracking limit on large raw-SQL model
- Dominant language
- Shell
- Stars
- 577
- Forks
- 93
- Avg merge
- 9h 46m
- Merged PRs (30d)
- 1
Description
### Describe the bug
`dbt_project_evaluator` v1.3.4 fails on dbt Fusion when `stg_nodes` evaluates a large raw-SQL model. The failure occurs in the package's `default__find_all_hard_coded_references` macro, at the `re.findall(...)` call that scans each model's full raw SQL against several broad regex patterns.
### Steps to reproduce
1. Have a large raw-SQL model in the project — in our case, `mart_merchant_transaction_features`, approximately 2,762 lines / 74 KB, containing a wide generated feature `SELECT` and a normal `FROM {{ ref(...) }}` CTE reference.
2. Run `dbt build --select stg_nodes` (or any invocation that triggers `dbt_project_evaluator`'s graph scan) under Fusion.
3. The failure occurs while `stg_nodes` iterates through the graph and calculates `hard_coded_references` for every SQL model — it reaches this specific model and the regex scan throws the error below.
Note that the package's exception/exclusion mechanism does not help avoid this failure, because exceptions are applied only after `stg_nodes` has already evaluated the regex.
### Expected results
The evaluator should complete the graph scan, or skip/report the specific model without failing the entire evaluator job.
### Actual results
The whole `dbt build` invocation fails; no evaluator output is produced for any model, not just the large one.
### Screenshots and log output
```text
Error executing regex: Max limit for backtracking count exceeded
```
Stack trace points to:
```text
macros/unpack/find_all_hard_coded_references.sql:242
re.findall(regex_pattern, model_raw_sql_no_comments)
```
### System information
**The contents of your `packages.yml` file:**
```yaml
packages:
- package: dbt-labs/dbt_project_evaluator
version: 1.3.4
```
**Which database are you using dbt with?**
- [ ] postgres
- [ ] redshift
- [x] bigquery
- [ ] snowflake
- [ ] trino/starburst
- [ ] other (specify: ____________)
**The output of `dbt --version`:**
```
dbt-fusion 2.0.0-preview.212
```
### Additional context
The regex patterns in `find_all_hard_coded_references` run over each model's entire raw SQL. On this model, they trigger Fusion's regex engine backtracking limit. Related but distinct: #581 reports a different Fusion incompatibility in this same macro (`match[1:]` failing with "value of type map cannot be sliced").
### Are you interested in contributing the fix?
No.
Contributor guide
Assessment
This issue has not been assessed yet.