dbt-labs / dbt-labs/dbt-external-tables

Allow configuration of `refresh_on_create` for Snowflake External Tables on AWS

Open
#348 1 comment 0 reactions 0 assignees View on GitHub
enhancement snowflake Stale triage
Dominant language
Jupyter Notebook
Stars
384
Forks
150
Avg merge
1h 25m
Merged PRs (30d)
1

Description

### Describe the feature

There's no way to set `refresh_on_create` to false. This property is useful to set for extremely large number of files in S3 paths that would breach the limits for auto refresh with this error.

```
Encountered an error while running operation: Database Error
001057 (0A000): Total size (>=1,073,742,112 bytes) for the list of file descriptors returned from the stage exceeded limit (1,073,741,824 bytes); Number of file descriptors returned is >=3,273,604. Please use a prefix in the stage location or pattern option to reduce the number of files.
```

### Describe alternatives you've considered
Not using this package and using DDL. Other alternative is of course to define external tables on more specific prefixes so as not to breach the file listing limits.

### Additional context
This feature is Snowflake on AWS specific.

### Who will this benefit?
Anyone having large external tables being moved to dbt.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.