dbt-labs / dbt-labs/docs.getdbt.com

Document catalog-qualified three-part identifiers for dbt-spark

Open
#9,867 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
215
Forks
1.2k
Avg merge
1d 15h
Merged PRs (30d)
130

Description

### Contributions

- [x] I have read the contribution docs and understand what is expected.

### Link to the page on docs.getdbt.com requiring updates

- https://docs.getdbt.com/docs/local/connect-data-platform/spark-setup
- https://docs.getdbt.com/reference/resource-configs/spark-configs

### What part(s) of the page would you like to see updated?

Update the dbt Core/dbt-spark documentation for catalog-qualified identifiers
tracked in dbt-labs/dbt-adapters#495.

The current Spark configuration page says users should never set `database`.
That guidance will become incorrect.

Please:

- Document `catalog` as the recommended profile alias for dbt's `database`
field.
- Explain that Spark `schema` represents the namespace; `namespace` is not a
separate profile key.
- Document that model and source configurations use dbt's canonical
`database` field to select another catalog.
- State that three-part naming is independent of the connection method and
applies to `session`, `thrift`, `http`, and `odbc`.
- Add a profile example:

```yaml
type: spark
method: session
host: localhost
catalog: my_catalog
schema: my_namespace
```

- Explain that this produces `my_catalog.my_namespace.table_name`.
- Clarify that `method: session` is illustrative; the connection method only
selects the transport and does not change identifier rendering.
- State that omitting `catalog` preserves two-part `schema.table` identifiers.
- Document the compatibility rule: an existing profile with matching
`database` and `schema` remains two-part; use explicit `catalog` when the
catalog and namespace intentionally have the same name.
- Note that `catalog` and `database` are aliases and should not both be set.
- Warn that existing model/source `database` or `+database` values, previously
ignored by dbt-spark, will begin selecting a catalog after this change.
- State that Spark itself must support multipart identifiers and the named
catalog must already be configured; dbt-spark does not install catalog
plugins.
- Replace the "Always `schema`, never `database`" section with the new
catalog/database guidance.
- Update only the dbt Core/dbt-spark content. Do not claim Fusion support
unless it is independently confirmed.

### Additional information

- https://github.com/dbt-labs/dbt-adapters/issues/495
- https://github.com/dbt-labs/dbt-adapters/issues/2129
- https://github.com/dbt-labs/dbt-adapters/issues/480
- https://github.com/dbt-labs/dbt-adapters/issues/480#issuecomment-2588413198

The implementation PR will link this issue.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.