dbt-labs / dbt-labs/spark-utils

spark__get_relations_by_pattern doesn't seem to work as expected

Open
#36 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
36
Forks
14
PR merge metrics
No merged PRs in 30d

Description

spark_utils==0.3.0

** Edit ** just realized the readme says that it doesn't have a shim for this, but it sure seems to.

Found this because I was trying to figure out [this codegen issue](https://github.com/dbt-labs/dbt-codegen/issues/145)

if I call

```
dbt run-operation generate_source --args '{schema_name: my_schema, database_name: my_database}'
```

I don't get anything back

But if I call

```
dbt run-operation generate_source --args '{"schema_name": "my_schema", "database_name": "my_database", "table_pattern": ""*""}'
```

I do get stuff back, but I get the target.database instead `my_database` (they have the same schema).
I think that's because [database=None](https://github.com/dbt-labs/spark-utils/blob/main/macros/dbt_utils/sql/get_relations_by_prefix.sql#L15)

I'm guessing I could do something [similar to this](https://github.com/dbt-labs/spark-utils/blob/main/macros/dbt_utils/sql/get_relations_by_prefix.sql#L15) in order to get the table_pattern problem. codegen defaults it to `'%'`

## Possible Workaround
I'm able to get it working if I add
```
{% set table_pattern = table_pattern|replace('%', '*') %}
{%- call statement('switch_database', fetch_result=False) %}
USE CATALOG {{database}}
{%- endcall -%}
```
to the top of [spark__get_relations_by_pattern](https://github.com/dbt-labs/spark-utils/blob/main/macros/dbt_utils/sql/get_relations_by_prefix.sql#L2)

But I don't know what kind of impact that might truly have. It's only the path I'm coming from.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with macros/dbt_utils/sql/get_relations_by_prefix.sql and reproduce the issue using the two dbt run-operation commands in the report. Compare the behavior of spark__get_relations_by_pattern with and without table_pattern, including the reported database value. Done means the relation lookup returns the expected objects for the requested schema and database without introducing unintended catalog-switching effects.

Written by the indexing model from the issue text.

Assessment

Tech stack
spark
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.