dbt-labs / dbt-labs/docs.getdbt.com
Document catalog-qualified three-part identifiers for dbt-spark
- Dominant language
- JavaScript
- Stars
- 215
- Forks
- 1.2k
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 130
Description
### Contributions
- [x] I have read the contribution docs and understand what is expected.
### Link to the page on docs.getdbt.com requiring updates
- https://docs.getdbt.com/docs/local/connect-data-platform/spark-setup
- https://docs.getdbt.com/reference/resource-configs/spark-configs
### What part(s) of the page would you like to see updated?
Update the dbt Core/dbt-spark documentation for catalog-qualified identifiers
tracked in dbt-labs/dbt-adapters#495.
The current Spark configuration page says users should never set `database`.
That guidance will become incorrect.
Please:
- Document `catalog` as the recommended profile alias for dbt's `database`
field.
- Explain that Spark `schema` represents the namespace; `namespace` is not a
separate profile key.
- Document that model and source configurations use dbt's canonical
`database` field to select another catalog.
- State that three-part naming is independent of the connection method and
applies to `session`, `thrift`, `http`, and `odbc`.
- Add a profile example:
```yaml
type: spark
method: session
host: localhost
catalog: my_catalog
schema: my_namespace
```
- Explain that this produces `my_catalog.my_namespace.table_name`.
- Clarify that `method: session` is illustrative; the connection method only
selects the transport and does not change identifier rendering.
- State that omitting `catalog` preserves two-part `schema.table` identifiers.
- Document the compatibility rule: an existing profile with matching
`database` and `schema` remains two-part; use explicit `catalog` when the
catalog and namespace intentionally have the same name.
- Note that `catalog` and `database` are aliases and should not both be set.
- Warn that existing model/source `database` or `+database` values, previously
ignored by dbt-spark, will begin selecting a catalog after this change.
- State that Spark itself must support multipart identifiers and the named
catalog must already be configured; dbt-spark does not install catalog
plugins.
- Replace the "Always `schema`, never `database`" section with the new
catalog/database guidance.
- Update only the dbt Core/dbt-spark content. Do not claim Fusion support
unless it is independently confirmed.
### Additional information
- https://github.com/dbt-labs/dbt-adapters/issues/495
- https://github.com/dbt-labs/dbt-adapters/issues/2129
- https://github.com/dbt-labs/dbt-adapters/issues/480
- https://github.com/dbt-labs/dbt-adapters/issues/480#issuecomment-2588413198
The implementation PR will link this issue.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.