dbt-labs / dbt-labs/dbt-adapters
[Bug] dbt-spark drops the catalog prefix when listing relations in a catalog-qualified schema (schema: my_catalog.my_namespace)
- Dominant language
- Python
- Stars
- 233
- Forks
- 362
- Avg merge
- 3d 22h
- Merged PRs (30d)
- 9
Description
### Is this a new bug?
- [x] I believe this is a new bug
- [x] I have searched the existing issues, and I could not find an existing issue for this bug
### Which packages are affected?
- [ ] dbt-adapters
- [ ] dbt-tests-adapter
- [ ] dbt-athena
- [ ] dbt-athena-community
- [ ] dbt-bigquery
- [ ] dbt-postgres
- [ ] dbt-redshift
- [ ] dbt-snowflake
- [x] dbt-spark
### Current Behavior
To reach a Spark v2 (DSv2) catalog, dbt-spark users have to put the catalog in the schema, because `database` is forced to `None` in `SparkCredentials.__post_init__`:
```yaml
schema: my_catalog.my_namespace
list_relations_without_caching then builds its relations from the namespace column returned by SHOW TABLES / SHOW TABLE EXTENDED. But Spark reports that column without the catalog prefix:
spark-sql> show tables in my_catalog.my_namespace;
namespace tableName isTemporary
my_namespace my_table false
_get_relation_information_using_describe takes that value verbatim (dbt-spark/src/dbt/adapters/spark/impl.py:185):
_schema, name, _ = row
...
table_name = f"{_schema}.{name}"
so _schema is my_namespace, not my_catalog.my_namespace. Three things go wrong as a result:
1. The auxiliary DESCRIBE targets the wrong table. It runs describe extended my_namespace.my_table, which resolves against the default catalog. The resulting DbtRuntimeError is swallowed and logged at debug only (impl.py:196-198), so the run looks clean. If a same-named namespace happens to exist in the default catalog, it is worse than a miss — DESCRIBE succeeds and returns metadata for an entirely different table.
2. Every cache lookup misses. The relation is created with schema="my_namespace" while dbt looks it up as my_catalog.my_namespace, so adapter.get_relation(...) returns None for tables that plainly exist. Incremental models take the create path instead of merging.
The same shape is present in _get_relation_information (impl.py:171-179), the SHOW TABLE EXTENDED path. I have not verified that path against a v2 catalog, since SHOW TABLE EXTENDED errors out for v2 tables anyway (SPARK-33393).
### Expected Behavior
Relations listed for `schema: my_catalog.my_namespace` should carry the schema dbt asked for, not the bare namespace Spark echoes back. Concretely:
1. The auxiliary statement should be `describe extended my_catalog.my_namespace.my_table`, so it resolves in the catalog the profile points at — and cannot silently hit a same-named table in the default catalog.
2. `information` should be populated, so views are typed as `RelationType.View` and `is_delta` / `is_iceberg` / `is_hudi` reflect the actual provider.
3. Listed relations should render as `my_catalog.my_namespace.my_table`, matching the name dbt uses for lookups, so the relation cache hits and an existing incremental model is merged into rather than rebuilt.
In short, `dbt run` against a catalog-qualified schema should behave exactly as it does against a plain one — the catalog prefix should be invisible to everything downstream of `list_relations_without_caching`.
### Steps To Reproduce
1. A Spark cluster with a DSv2 catalog configured, e.g. an Iceberg catalog named `my_catalog`:
SET spark.sql.catalog.my_catalog=org.apache.iceberg.spark.SparkCatalog;
SET spark.sql.catalog.my_catalog.type=hadoop;
SET spark.sql.catalog.my_catalog.warehouse=hdfs:///hdfs_path;
2. Create a namespace and a v2 table in it:
```sql
create namespace if not exists my_catalog.my_namespace;
create table my_catalog.my_namespace.seed_table (id int) using iceberg;
insert into my_catalog.my_namespace.seed_table values (1);
3. Confirm Spark drops the catalog from the namespace column — this is the root cause, and it reproduces without dbt at all:
show tables in my_catalog.my_namespace;
-- namespace = my_namespace (not my_catalog.my_namespace)
4. profiles.yml:
`my_project:
target: dev
outputs:
dev:
type: spark
method: thrift
host:
port:
schema: my_catalog.my_namespace`
5. A single incremental model, models/my_table.sql:
{{ config(materialized='iiceberg') }} select 1 as id
6. dbt run — succeeds, creates my_catalog.my_namespace.my_table.
7. dbt --debug run again, and observe two symptoms:
- The debug log contains describe extended my_namespace.my_table — the catalog prefix is missing — followed by Error while retrieving information about my_namespace.my_table.
- The model is created from scratch rather than merged into, even though the table exists and this is not a --full-refresh.
### Relevant log output
```shell
show tables in prod_dm_dbt.crm
return
namespace
---------
crm
```
### Environment
```markdown
- OS: CentOS 7 Linux
- Python: 3.11.0
- dbt-adapters: 1.7.0
- dbt-spark : 1.10.0a1
```
### Additional Context
````python
def _get_relation_information_using_describe(
self, relation: BaseRelation, row: "agate.Row"
) -> RelationInfo:
"""Relation info fetched using SHOW TABLES and an auxiliary DESCRIBE statement"""
try:
_, name, _ = row
except ValueError:
raise DbtRuntimeError(
f'Invalid value from "show tables ...", got {len(row)} values, expected 3'
)
table_name = f"{relation.schema}.{name}"
try:
table_results = self.execute_macro(
DESCRIBE_TABLE_EXTENDED_MACRO_NAME, kwargs={"table_name": table_name}
)
except DbtRuntimeError as e:
logger.debug(
f"Error while retrieving information about {table_name}: {e.msg}"
)
table_results = AttrDict()
information = ""
for info_row in table_results:
info_type, info_value, _ = info_row
if not info_type.startswith("#"):
information += f"{info_type}: {info_value}\n"
return relation.schema, name, information
````
currently using relation.schema instead of _schema as a workaround
Contributor guide
Assessment
This issue has not been assessed yet.