dbt-labs / dbt-labs/spark-utils
Add support for get_relations_by* / get_tables_by* using information_schema in Unity Catalog
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 36
- Forks
- 14
- PR merge metrics
- No merged PRs in 30d
Description
Databricks and Unity Catalog now supports information_schema https://docs.databricks.com/sql/language-manual/sql-ref-information-schema.html and so it should now be possible to support some of the missing shims listed in the readme and as one of the tasks in https://github.com/dbt-labs/spark-utils/issues/7:
The missing shims are:
- dbt_utils.get_relations_by_prefix_sql
- dbt_utils.get_tables_by_pattern_sql
- dbt_utils.get_tables_by_prefix
- dbt_utils.get_tables_by_pattern
I will be willing to try and contribute these but will need guidance on how to define the logic that only allows these to be compatible if you are using >=dbt-databricks==1.1.1 and have Unity Catalog enabled.
Thanks
Tahir
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the missing-shim list in the README and the related spark-utils issue #7. Review the four requested get_relations_by* and get_tables_by* shims and the stated dbt-databricks >=1.1.1 and Unity Catalog compatibility requirement; done means those shims work through information_schema only under the supported conditions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- sql
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100