DB engine for pandas: sql.connect or sqlalchemy

Open

#476 opened on Nov 27, 2024 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty: 4/5
Estimated time: 3-5 days
Newbie friendliness: 25/100

Issue type: Documentation
Clarity: Needs clarification
Activity status: Stale
Tech stack: pandas, python, sql, sqlalchemy
Domain: data, databases

Research direction

The issue compares databricks.sql.connect with SQLAlchemy and pandas.read_sql, including CloudFetch and possible serialization overhead. Start by reviewing the linked closed pandas-related pull requests and the pandas.read_sql integration. Done means documenting the recommended API, performance implications, and whether both paths retain CloudFetch.

Written by the indexing model from the issue text.

Description

Hello,

I was wondering what's the best practice for using this package with pandas.

It's possible to create a databricks.sql.connect and pass it to pandas.read_sql. This works however it raises

UserWarning: pandas only supports SQLAlchemy connectable (engine/connection) or database string URI or sqlite3 DBAPI2 
connection. Other DBAPI2 objects are not tested. Please consider using SQLAlchemy.

Alternatively it's possible to use SQLAlchemy with a databricks:// URL and pass that to pandas. Doesn't it mean an extra serialization step performance wise though?

What's the recommended way, in particular regarding performance? Would both use CloudFetch for larger queries? I see there are some fixes/improvements done for pandas done in PRs so which API should be used to benefit from those?

Thanks!

cc @kravets-levko

Dominant language: Python
Stars: 233
Forks: 152
Avg merge: 21h 5m
Merged PRs (30d): 10

Contributor guide

Open the contributing guide

First steps

Read the whole issue, then the project's contributing guide.
Comment on the issue to say you are picking it up — it saves two people doing the same work.
Fork the repository and make your change on a branch.
Open a pull request that references the issue number.

Similar issues

Update contributor page in the docs Open

fix: inaccuracy ⚠️

Difficulty 2/5 1-3 hours Newbie friendliness 72/100

uabrc/uabrc.github.io#1255 · 1 comment · Opened Sep 21, 2026
melange/apko: attest-build-provenance always fails on fork PRs (missing id-token) Open

Difficulty 2/5 1-3 hours Newbie friendliness 84/100

ethereum-optimism/factory#64 · Opened Sep 21, 2026
Type stubs omit the `connection` keyword on module-level `read_json` and `write_csv` Open

Difficulty 2/5 1-3 hours Newbie friendliness 90/100

duckdb/duckdb-python#627 · Opened Sep 21, 2026
custom cli groups do not show up when using the --help flag Open

Difficulty 2/5 1-3 hours Newbie friendliness 68/100

geobeyond/potto#73 · Opened Sep 21, 2026
Add link for tutorial Open

documentation

Difficulty 1/5 Under an hour Newbie friendliness 78/100

Qiskit/qiskit-addon-sqd#376 · Opened Sep 21, 2026

More Python issues