dbt-labs / dbt-labs/dbt-external-tables
Allow configuration of `refresh_on_create` for Snowflake External Tables on AWS
- Dominant language
- Jupyter Notebook
- Stars
- 384
- Forks
- 150
- Avg merge
- 1h 25m
- Merged PRs (30d)
- 1
Description
### Describe the feature
There's no way to set `refresh_on_create` to false. This property is useful to set for extremely large number of files in S3 paths that would breach the limits for auto refresh with this error.
```
Encountered an error while running operation: Database Error
001057 (0A000): Total size (>=1,073,742,112 bytes) for the list of file descriptors returned from the stage exceeded limit (1,073,741,824 bytes); Number of file descriptors returned is >=3,273,604. Please use a prefix in the stage location or pattern option to reduce the number of files.
```
### Describe alternatives you've considered
Not using this package and using DDL. Other alternative is of course to define external tables on more specific prefixes so as not to breach the file listing limits.
### Additional context
This feature is Snowflake on AWS specific.
### Who will this benefit?
Anyone having large external tables being moved to dbt.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.