dbt-labs / dbt-labs/dbt

[1.x Bug] dbt docs generate ignores --select/--exclude in catalog queries once selected relations exceed MAX_SCHEMA_METADATA_RELATIONS (100)

Open
#15,462 1 comment 0 reactions 0 assignees View on GitHub
triage
Dominant language
Rust
Stars
13.8k
Forks
2.6k
Avg merge
21h 31m
Merged PRs (30d)
56

Description

### Is this a new bug in dbt-core?

- [x] I believe this is a new bug in dbt-core
- [x] I have searched the existing issues, and I could not find an existing issue for this bug

### Current Behavior

`docs generate --exclude source:foo` still queries `foo`'s schema when the selected set has
more than 100 relations. Below 100 it works fine. The excluded source lives in a catalog our
workspace can't access, so the catalog query blows up and `docs generate` exits non-zero
(catalog.json still gets written):

```
PERMISSION_DENIED: Catalog 'inaccessible_catalog' is not accessible in current workspace
```

Looks like the >100 case falls back to `get_filtered_catalog`'s full path, which builds
queries from every schema in the manifest instead of the selected relations. Selection only
gets applied afterwards as a filter on the result rows, so the query already ran.

### Expected Behavior

Selection should scope the catalog queries no matter how many relations are selected. A schema
with nothing selected in it shouldn't be queried at all — same as it works under 100.

### Steps To Reproduce

1. Create 101 trivial models in one schema (to get over the 100 threshold):
```bash
for i in $(seq -w 0 100); do echo 'select 1 as id' > "models/model_${i}.sql"; done
```
2. Add two sources — one in a reachable catalog, one in a catalog you can't access:
```yaml
sources:
- name: source_present
database: my_catalog
schema: my_schema
tables: [{name: present_table}]
- name: source_excluded
database: inaccessible_catalog
schema: excluded_schema
tables: [{name: excluded_table}]
```
3. `dbt docs generate --exclude source:source_excluded` → fails, queries `inaccessible_catalog`.
4. Delete 11 models so the selected set is <100, run the same command → passes, never touches
`inaccessible_catalog`.

### Relevant log output

```
# >100 relations: a catalog connection to the excluded schema is opened and USE CATALOG fails
Building catalog
Databricks adapter: DatabricksDBTConnection(session-id=None, name=('inaccessible_catalog', 'excluded_schema')) - Creating connection
On ('inaccessible_catalog', 'excluded_schema'): /* ... */
USE CATALOG `inaccessible_catalog`
Databricks adapter: Exception while trying to execute query
/* ... */
USE CATALOG `inaccessible_catalog`
: PERMISSION_DENIED: Catalog 'inaccessible_catalog' is not accessible in current workspace
Encountered an error while generating catalog: Database Error
PERMISSION_DENIED: Catalog 'inaccessible_catalog' is not accessible in current workspace
Wrote artifact CatalogArtifact to ./target/catalog.json
dbt encountered 1 failure while writing the catalog
# exit 1
```

```
# <100 relations: only the selected schema is queried, exits 0
On ('my_catalog', 'my_schema'): /* ... */
SHOW TABLE EXTENDED IN `my_catalog`.`my_schema` LIKE 'model_000|...|present_table'
# no connection to inaccessible_catalog is ever opened — exit 0
```

### Environment

```markdown
- OS: macOS 26.5
- Python: 3.11.14
- dbt: 1.11.8 (dbt-databricks 1.11.7)
```

### Which database adapter are you using with dbt?

other (databricks) — but this is dbt-core logic (`get_filtered_catalog` / the 100-relation
threshold), so it should hit any adapter with the `SchemaMetadataByRelations` capability.

### Additional Context

Probably the still-broken part of #6014 / #7259 above the 100-relation cutoff.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.