apache / apache/arrow-adbc

ADBC Driver Optimization for Power BI Integration

Open
#3,169 4 comments 0 reactions 0 assignees View on GitHub
Type: question
Dominant language
C#
Stars
627
Forks
217
Avg merge
17h
Merged PRs (30d)
57

Description

### What would you like help with?

I have a customer who is experiencing slow data transfer from Snowflake to Power BI using ODBC driver version 3.2.2, despite the query execution in Snowflake performing well. They are exploring the new ADBC driver recommended by Microsoft to potentially improve performance. However, the issue remains. Fetching large result sets (e.g., 160M rows) via ADBC is still slow due to small default fetch batch sizes (~10K rows).

They noticed that the extraction is happening in cursor mode, in chunks e.g., data fetching is in small batches (10000 rows) which took 20 min to fetch 160MM rows.

As such, the customer has the following question. Kindly provide any guidance you can.
1. What are the best practices for using the ADBC driver with Power BI?
2. Are there any configurations to optimize the ADBC driver further?
3. The environment variable needed for the ADBC driver (http_proxy) is generic and impacts other applications. Is there a way to rename this? Can we implement an ADBC-specific proxy configuration to avoid conflicts with other connectors?
4. What can be done to maximize throughput (batching, parallelism, or other configurations)?
5. Do we have any ability to configure the chunk size or exclude, if possible?
6. Are there any Snowflake-provided benchmarks or statistics on data load timing we can reference?
7. Are there any stats on the data load timing from Snowflake team that can be referred for the benchmarking ?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.