dropbox / dropbox/PyHive

Slow performance reading large Hive table in comparison with RJDBC

Open
#357 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
545
PR merge metrics
No merged PRs in 30d

Description

I'm trying to read a large table from Hive in python using pyhive, the table has about 16 millions of rows. But it is taking about to **33 minutes**. When I read the same table in R with RJDBC it takes about **13 minutes** to read the whole table. Here is my code.

``` R
library(RJDBC)

driver <- try(JDBC("org.apache.hive.jdbc.HiveDriver", paste0(jar_dir, '/hive-jdbc-3.1.2-standalone.jar')))
con_hive <- RJDBC::dbConnect(driver, "jdbc:hive2://hive_ip:10000/dev_perm")
query <- "SELECT * FROM mi table WHERE periodo='2020-02-01'"
replica_data <- dbGetQuery(con_hive, query)
```
And in python my code is

``` python
import pyhive
conn = hive.Connection(host=ip_hive)
curs = conn.cursor()
cursor.execute("SELECT * FROM mi table WHERE periodo='2020-02-01'")
results = pd.DataFrame(cursor.fetchall(), columns=[desc[0] for desc in cursor.description])
```
I already tried to set multiple cursor.arraysize in python but it doesn't improve performace and also I notice when I set a arraysize greater than 10000 hive ignores it and set 10000. The default value is 1000.

What can I do to improve my performace reading Hive tables in python?

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.