dbt-labs / dbt-labs/dbt

info schema: fold warehouse catalog columns and stats into the per-resource-type tables

Open
#16,044 0 comments 0 reactions 0 assignees View on GitHub
area:engine
Dominant language
Rust
Stars
13.8k
Forks
2.6k
Avg merge
21h 31m
Merged PRs (30d)
56

Description

Follow-up from dbt-labs/fs#13498.

#15850 says to remove `dbt.catalog_tables`, `dbt.catalog_stats` and `dbt.column_stats` and combine their columns into the tables that already share their grain — `dbt.models`, `dbt.seeds`, `dbt.snapshots` and `dbt.node_columns` — populated only when the data has been fetched from the warehouse, as was already done for `catalog_columns`.

The first PR did the **remove** half. The **combine** half is still open, because it isn't a column projection:

- `run/catalog_stats` is an EAV table — `stat_id` / `stat_label` / `stat_value`, with values coerced to strings. Folding it in means **un-pivoting** on a known set of stat ids and then joining, not renaming a column.
- The two sides arrive at different times: the `dbt.models` row exists after parse, while catalog data needs a `Run`/`Build` plus a warehouse round-trip. So the fold needs carry-forward semantics on the per-resource-type files, otherwise a later parse-only invocation blanks the columns that a previous run populated.
- `run/catalog_stats` has no `table_comment`; the legacy path hardcoded it to NULL.

### Columns to fold

Onto `dbt.models` / `dbt.seeds` / `dbt.snapshots`, from `catalog_tables` and `catalog_stats`:
`table_type`, `table_owner`, `table_comment`, `row_count`, `bytes`, `last_modified`

Onto `dbt.node_columns`, from `column_stats`:
`row_count`, `distinct_count`, `null_pct`, `min_value`, `max_value`, `avg_value`, `std_value`, `q25`, `q50`, `q75`, `top_values`

### Note on shape stability

These columns are **absent** from the information schema today rather than present-and-null. Adding them later is an additive change, but it does mean a consumer written against the current shape sees new columns appear. If that matters, declaring them up front as null-filled is cheap — `fs/sa/crates/dbt-index-core/src/info_schema/schema.rs` supports declaring a typed column with no source for exactly this case.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.