Upgrade Databricks SQL Alchemy version to avoid issues with metadata operations
- Dominant language
- Python
- Stars
- 74.8k
- Forks
- 18.3k
- Avg merge
- 2d 5h
- Merged PRs (30d)
- 685
Description
### Bug description
Hi there,
Some of the users are hitting an issue on Superset v6 using the [databricks python connector](https://superset.apache.org/user-docs/databases/supported/databricks/)
The metadata discovery (table browser) is currently broken when using catalogs and/or schemas containing hyphens (which is very common in Databricks Unity Catalog). Superset fails to fetch the table list because the generated SQL is not properly quoted.
For example, it currently generates:
`SHOW TABLES FROM my-staging-catalog.my-poc-schema`
Instead of:
``SHOW TABLES FROM `my-staging-catalog`.`my-poc-schema` ``
This triggers a [INVALID_IDENTIFIER] error because the hyphen is interpreted as a subtraction operator rather than part of the identifier.
```
[INVALID_IDENTIFIER] The unquoted identifier my-staging-catalog is invalid and must be back quoted as: `my-staging-catalog`.
Unquoted identifiers can only contain ASCII letters ('a' - 'z', 'A' - 'Z'), digits ('0' - '9'), and underbar ('_').
```
This was fixed in SQL Alchemy in v2.0.4 but the repo has pinned v1.0.5
Can we please update to point to the recent version of Databricks SQL Alchemy?
### Screenshots/recordings
_No response_
### Superset version
master / latest-dev
### Python version
3.9
### Node version
Not applicable
### Browser
Chrome
### Additional context
_No response_
### Checklist
- [x] I have searched Superset docs and Slack and didn't find a solution to my problem.
- [x] I have searched the GitHub issue tracker and didn't find a similar bug report.
- [ ] I have checked Superset's logs for errors and if I found a relevant Python stacktrace, I included it here as text in the "additional context" section.
Contributor guide
Research direction
Find the dependency pin for the Databricks SQLAlchemy connector and trace the metadata-discovery path used for table browsing. Verify the change against catalogs and schemas containing hyphens; done means the dependency is updated to a version that generates correctly quoted SHOW TABLES queries without breaking existing metadata operations.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, sqlalchemy
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 70/100