dropbox / dropbox/PyHive

acquireReadWriteLocks takes more time while running Hive query by PyHive

Open
#255 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
545
PR merge metrics
No merged PRs in 30d

Description

Hi,
I am trying to run my query through PyHive, It takes more time for the execution and I have observed that one of the **acquireReadWriteLocks** takes more time in the execution plan. If that is case Could you please help me how to resolve it.

acquireReadWriteLocks= 3m 37s 10ms (in Total Query execution Duration is 2m 42s 924ms)
acquireReadWriteLocks= 3m 43s 47ms (in Total Query execution Duration is 3m 59s 587ms)

The same Hive Query is getting completed in less than 10 seconds if I try through Hive Beeline

PyHive script to connect Hive.
-----------------------------------
from pyhive import hive
import pandas as pd

hive_conn = hive.connect('ip-address',username='USER-NAME', password='PASSWORD',auth='XXXX')
query="select * from HIVE_DB.TABLE_NAME where date >= '2011-01-01' and date <= '2011-01-02' "
df = pd.read_sql_query(query,con=hive_conn)
hive_conn.close()

Note:
Table is .orc file-format, partitioned by date column

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.