dbt-labs / dbt-labs/dbt-adapters

[Bug] dbt run against spark profile checks for database even in unrelated sources

Open
#849 2 comments 1 reaction 0 assignees View on GitHub
type:bug
Dominant language
Python
Stars
233
Forks
362
Avg merge
3d 22h
Merged PRs (30d)
9

Description

### Is this a new bug?

- [x] I believe this is a new bug
- [x] I have searched the existing issues, and I could not find an existing issue for this bug

### Which packages are affected?

- [ ] dbt-adapters
- [ ] dbt-tests-adapter
- [ ] dbt-athena
- [ ] dbt-athena-community
- [ ] dbt-bigquery
- [ ] dbt-postgres
- [ ] dbt-redshift
- [ ] dbt-snowflake
- [x] dbt-spark

### Current Behavior

sources.yml
```
- name: spark
description: glue default database
schema: default
tables:
- name: fact_customer_item_dly
description: fact_customer_item_dly
```
profiles.yml
```
sparkprofile:
target: dev
outputs:
dev:
type: spark
method: thrift
host: localhost
port: 10001
schema: dev
```
spark_test.sql
```
{{ config(
materialized='table',
file_format='parquet'
) }}

select count(*) from {{ source('spark' ,'fact_customer_item_dly') }}
```

`.venv/bin/dbt run --select spark_test.sql --profile sparkprofile` errors with Cannot set database in spark!

### Expected Behavior

With the above sources.yml, profiles.yml `.venv/bin/dbt run --select spark_test.sql --profile sparkprofile` should run without any error as source with name spark do not have any database specified

### Steps To Reproduce

Files sources.yml and profiles.yml should be as above

### Relevant log output

```shell
N/A
```

### Environment

```markdown
- OS: MACOS
- Python: 3.11.11
- dbt-spark: 1.8.0
```

### Additional Context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.