apache / apache/superset

Upgrade Databricks SQL Alchemy version to avoid issues with metadata operations

Open Beginner friendly
#38,658 5 comments 1 reaction 0 assignees View on GitHub
data:connect:databricks dependencies:python validation:required
Dominant language
Python
Stars
74.8k
Forks
18.3k
Avg merge
2d 5h
Merged PRs (30d)
685

Description

### Bug description

Hi there,

Some of the users are hitting an issue on Superset v6 using the [databricks python connector](https://superset.apache.org/user-docs/databases/supported/databricks/)

The metadata discovery (table browser) is currently broken when using catalogs and/or schemas containing hyphens (which is very common in Databricks Unity Catalog). Superset fails to fetch the table list because the generated SQL is not properly quoted.

For example, it currently generates:

`SHOW TABLES FROM my-staging-catalog.my-poc-schema`

Instead of:

``SHOW TABLES FROM `my-staging-catalog`.`my-poc-schema` ``
This triggers a [INVALID_IDENTIFIER] error because the hyphen is interpreted as a subtraction operator rather than part of the identifier.

```
[INVALID_IDENTIFIER] The unquoted identifier my-staging-catalog is invalid and must be back quoted as: `my-staging-catalog`.
Unquoted identifiers can only contain ASCII letters ('a' - 'z', 'A' - 'Z'), digits ('0' - '9'), and underbar ('_').
```

This was fixed in SQL Alchemy in v2.0.4 but the repo has pinned v1.0.5
Can we please update to point to the recent version of Databricks SQL Alchemy?

### Screenshots/recordings

_No response_

### Superset version

master / latest-dev

### Python version

3.9

### Node version

Not applicable

### Browser

Chrome

### Additional context

_No response_

### Checklist

- [x] I have searched Superset docs and Slack and didn't find a solution to my problem.
- [x] I have searched the GitHub issue tracker and didn't find a similar bug report.
- [ ] I have checked Superset's logs for errors and if I found a relevant Python stacktrace, I included it here as text in the "additional context" section.

Contributor guide

Open the contributing guide

Research direction

Find the dependency pin for the Databricks SQLAlchemy connector and trace the metadata-discovery path used for table browsing. Verify the change against catalogs and schemas containing hyphens; done means the dependency is updated to a version that generates correctly quoted SHOW TABLES queries without breaking existing metadata operations.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, sqlalchemy
Domain
databases
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
70/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.