dbt-labs / dbt-labs/spark-utils
spark__get_relations_by_pattern doesn't seem to work as expected
- Dominant language
- Python
- Stars
- 36
- Forks
- 14
- PR merge metrics
- No merged PRs in 30d
Description
spark_utils==0.3.0
** Edit ** just realized the readme says that it doesn't have a shim for this, but it sure seems to.
Found this because I was trying to figure out [this codegen issue](https://github.com/dbt-labs/dbt-codegen/issues/145)
if I call
```
dbt run-operation generate_source --args '{schema_name: my_schema, database_name: my_database}'
```
I don't get anything back
But if I call
```
dbt run-operation generate_source --args '{"schema_name": "my_schema", "database_name": "my_database", "table_pattern": ""*""}'
```
I do get stuff back, but I get the target.database instead `my_database` (they have the same schema).
I think that's because [database=None](https://github.com/dbt-labs/spark-utils/blob/main/macros/dbt_utils/sql/get_relations_by_prefix.sql#L15)
I'm guessing I could do something [similar to this](https://github.com/dbt-labs/spark-utils/blob/main/macros/dbt_utils/sql/get_relations_by_prefix.sql#L15) in order to get the table_pattern problem. codegen defaults it to `'%'`
## Possible Workaround
I'm able to get it working if I add
```
{% set table_pattern = table_pattern|replace('%', '*') %}
{%- call statement('switch_database', fetch_result=False) %}
USE CATALOG {{database}}
{%- endcall -%}
```
to the top of [spark__get_relations_by_pattern](https://github.com/dbt-labs/spark-utils/blob/main/macros/dbt_utils/sql/get_relations_by_prefix.sql#L2)
But I don't know what kind of impact that might truly have. It's only the path I'm coming from.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with macros/dbt_utils/sql/get_relations_by_prefix.sql and reproduce the issue using the two dbt run-operation commands in the report. Compare the behavior of spark__get_relations_by_pattern with and without table_pattern, including the reported database value. Done means the relation lookup returns the expected objects for the requested schema and database without introducing unintended catalog-switching effects.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- spark
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100