dbt-labs / dbt-labs/dbt

catalogs.yml: don't require external_volume for horizon catalogs when storage is Snowflake-managed

Open
#15,921 0 comments 0 reactions 1 assignee Claimed by @VersusFacit View on GitHub
area:engine proj:catalog-v2 type:feature
Dominant language
Rust
Stars
13.8k
Forks
2.6k
Avg merge
21h 31m
Merged PRs (30d)
56

Description

Part of #15679 (catalogs v2 GA) — open design question, not yet gaveled.

\`HORIZON_SNOWFLAKE_FIELDS\` marks \`external_volume\` \`.required().non_empty()\` ([\`dbt_catalogs_v2.rs:104-107\`](https://github.com/dbt-labs/fs/blob/main/fs/sa/crates/dbt-schemas/src/schemas/dbt_catalogs_v2.rs)) for any \`type: horizon\` catalog's \`snowflake\` block — even when the value is just \`SNOWFLAKE_MANAGED\`, which the runtime materialization path already defaults to on its own ([\`catalog_relation_v2.rs:590-594\`](https://github.com/dbt-labs/fs/blob/main/fs/sa/crates/dbt-adapter/src/catalog_relation/catalog_relation_v2.rs)). So a customer using fully Snowflake-managed Iceberg storage (no S3 bucket of their own) still has to write a line in \`catalogs.yml\` just to say "use the default":

\`\`\`yaml
external_volume: "{{ env_var('SNOWFLAKE_EXTERNAL_VOLUME', 'SNOWFLAKE_MANAGED') }}"
\`\`\`

Could the schema allow omitting \`external_volume\` for \`horizon\` (and other \`BUILT_IN\`-style) catalogs, falling through to the same \`SNOWFLAKE_MANAGED\` default the runtime path already has? That'd make the zero-S3-setup path for Horizon-direct writes genuinely zero-config in \`catalogs.yml\`, not just zero-env-var.

Contrast: \`iceberg_rest\`/\`unity\`/\`glue\` (\`LINKED_SNOWFLAKE_FIELDS\`) correctly *forbid* \`external_volume\` since Snowflake isn't the storage owner there. \`horizon\` is the one case where Snowflake *is* the owner and could plausibly stay silent about it.

Not urgent — flagging alongside the other open catalogs v2 design questions.

Repro context: [dataders/dbt_aws_cloud_cost@horizon-only-repro](https://github.com/dataders/dbt_aws_cloud_cost/tree/horizon-only-repro)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.