Databricks: column-level lineage/catalog.json missing all models in a schema with >50 relations
- Dominant language
- Rust
- Stars
- 13.8k
- Forks
- 2.6k
- Avg merge
- 21h 31m
- Merged PRs (30d)
- 56
Description
## Summary
On Databricks, `catalog.json` (and therefore column-level lineage in dbt Explorer) silently omits **every** model in a schema once that schema has more than 50 relations. The run completes successfully with no errors or warnings; the catalog is simply missing those models.
Originally reported as [FUSCSE-38] (internal): a customer's `sap_dbt` project has 70 models in schema `silver_sap`. None of them appear in `catalog.json`, while a package (`dbt_project_evaluator`, 27 models in `silver_sap_evaluator`) and seeds (3, in `silver_sap_metadata`) in the same run appear correctly with full column data.
## Root cause
`crates/dbt-main/src/dbt_lib.rs`, function `fetch_catalog_data`, forks based on how many relations are in a schema:
```rust
let jinja_result: FsResult = if relation_as_values.len() > 50 {
let args = vec![
Value::from_serialize(db_schema),
Value::from_serialize(vec![schema.clone()]), // schema name as list[string]
];
get_catalog_by_relations(&jinja_env_clone, "get_catalog", ...)
} else {
let args = vec![
Value::from_serialize(db_schema),
Value::from_serialize(relation_as_values.clone()), // actual relation objects
];
get_catalog_by_relations(&jinja_env_clone, "get_catalog_relations", ...)
}
```
- Schemas with **≤50 relations** go through `get_catalog_relations`, passed real relation objects.
- Schemas with **>50 relations** go through `get_catalog` instead (to avoid an enormous per-relation `OR` chain), passed the bare schema name as `list[string]`, matching the documented signature on `get_catalog` in `crates/dbt-loader/src/dbt_macro_assets/dbt-adapters/macros/adapters/metadata.sql`:
```
-- funcsign: (relation, list[string]) -> agate_table
{% macro get_catalog(information_schema, schemas) -%}
```
The Databricks override of the schema where-clause builder does not honor this contract. `crates/dbt-loader/src/dbt_macro_assets/dbt-databricks/macros/adapters/catalog.sql`:
```jinja
{% macro databricks__get_catalog_schemas_where_clause_sql(catalog, schemas) -%}
WHERE table_catalog = '{{ catalog|lower }}' AND (
{%- for relation in schemas -%}
table_schema = '{{ relation[1]|lower }}'{%- if not loop.last %} OR {% endif -%}
{%- endfor -%})
{%- endmacro %}
```
This was written for the old classic-dbt-core convention where `schemas` was a set of `(database, schema)` tuples, so `relation[1]` pulled the schema out of the tuple. Since `schemas` is actually `list[string]` here, the loop yields `relation = "silver_sap"` (a plain string), and `relation[1]` indexes into the *string's characters* rather than a tuple, e.g. `"silver_sap"[1]` = `"i"` (0-indexed: s-**i**-l-v-e-r...).
Confirmed directly in the customer's debug log: the actual query sent to the warehouse was
```sql
WHERE table_catalog = 'kdp2_prod' AND (table_schema = 'i')
```
instead of `table_schema = 'silver_sap'`. That query correctly returns zero rows (no schema named `i` exists), so the whole schema is silently dropped from the catalog with no error.
**Contrast with Snowflake's equivalent macro** (`crates/dbt-loader/src/dbt_macro_assets/dbt-snowflake/macros/catalog.sql`), which correctly treats `schemas` as `list[string]`:
```jinja
{% macro snowflake__get_catalog_schemas_where_clause_sql(schemas) -%}
where ({%- for schema in schemas -%}
({{ snowflake__catalog_equals('table_schema', schema) }}){%- if not loop.last %} or {% endif -%}
{%- endfor -%})
{%- endmacro %}
```
Databricks is the only adapter in the repo with the `relation[1]` pattern.
## Reproduction
1. Create a Databricks project with a schema containing more than 50 models (any materialization, any SQL).
2. Run `dbt build --write-catalog` (or `dbt compile --write-catalog`).
3. Inspect the resulting `catalog.json`: every model in that schema is missing, with no error or warning anywhere in run output or debug logs at `info` level. At `debug` level, the catalog-fetch query for that schema will show a single-character `table_schema` filter instead of the real schema name.
Schemas with 50 or fewer relations in the same run are unaffected.
## Suggested fix
In `databricks__get_catalog_schemas_where_clause_sql`, change `relation[1]` to `relation` (and rename the loop variable from `relation` to `schema` to make the intent unambiguous and prevent recurrence).
## Impact
Silent and version-independent: reproduced identically on `2.0.0-preview.196` (2026-07-07) and `2.0.0-preview.202` (2026-07-28), so this is not something that was masked by an unrelated bugfix in between. Likely affects any Databricks customer with a schema of more than 50 relations using `--write-catalog` or dbt Explorer's column-level lineage.
[FUSCSE-38]: https://dbtlabs.atlassian.net/browse/FUSCSE-38?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ
Contributor guide
Assessment
This issue has not been assessed yet.