Appending a local Pandas dataframe to a Delta table in Azure is slow

Open
#450 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
35/100
Issue type
Bug
Clarity
Needs clarification
Activity status
Stale
Tech stack
azure, pandas, python, sqlalchemy
Domain
databases, performance

Research direction

No repository file or test is named. Start by reproducing the supplied pandas DataFrame.to_sql example with databricks-sql-connector 3.4.0 against the stated SQL warehouse, then trace the SQLAlchemy and connector write path to locate the per-row delay. Done means identifying a supported faster write path or documenting the limitation and its cause.

Written by the indexing model from the issue text.

Description

I followed the instructions on this page to create a SQLAlchemy engine and used it with the Pandas to_sql() method. It's taking around 2 seconds to append one data point to a Delta table in Azure Databricks, and it seems to be scaling linearly. A dataframe containing 1 column and 10 rows is taking ~20 seconds to push to Azure.

Is there a way to make writing back to Databricks from a local machine faster?

Example code:

from sqlalchemy import create_engine
import pandas as pd
import os

server = 'my_server.azuredatabricks.net'
http_path = "/sql/1.0/warehouses/my_warehouse"
access_token = "MY_TOKEN"
catalog = "my_catalog"
schema = "my_schema"

if "NO_PROXY" in os.environ:
    os.environ["NO_PROXY"] = os.environ["NO_PROXY"] + "," + server
else:
    os.environ["NO_PROXY"] = server

if "no_proxy" in os.environ:
    os.environ["no_proxy"] = os.environ["no_proxy"] + "," + server
else:
    os.environ["no_proxy"] = server

engine = create_engine(f"databricks://token:{access_token}@{server}?" +
        f"http_path={http_path}&catalog={catalog}&schema={schema}"
)

df = pd.DataFrame({'name' : ['User 1', 'User 2', 'User 3', User 4', 'User 5', 'User 6', User 7', 'User 8', 'User 9', 'User 10']})

# The next command takes around 20 seconds to complete
df.to_sql(name='my_test_table', con=engine, if_exists='append', index=False)

Local specs:
databricks-sql-connector==3.4.0
pandas==2.2.2
Python 3.10.14

Azure specs:
Databricks SQL warehouse cluster
Runtime==13.3 LTS

Dominant language
Python
Stars
233
Forks
152
Avg merge
21h 5m
Merged PRs (30d)
10

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from databricks/databricks-sql-python

All issues in databricks/databricks-sql-python

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.