dropbox / dropbox/PyHive

Possible DB-API fetchall() performance improvements

Open
#253 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
545
PR merge metrics
No merged PRs in 30d

Description

Currently the cursor implementation for [fetchall](https://github.com/dropbox/PyHive/blob/65076bbc8697a423b438dc03e928a6dff86fd2cb/pyhive/common.py#L129) indirectly iterates over the result set via `fetchone` rather than directly iterating over the result set similar to prestodb's python [DB-API](https://github.com/prestodb/presto-python-client/blob/master/prestodb/dbapi.py#L306).

I'm not certain whether it's viable to use this approach, however I mocked up a basic example which shows that the proposed solution is about 10x faster.

```python
import timeit

def current():
class Cursor:
def __init__(self):
self._iterator = iter(range(10000))

def fetchone(self):
try:
return next(self._iterator)
except StopIteration:
return None

def fetchall(self):
return list(iter(self.fetchone, None))

cursor = Cursor()
cursor.fetchall()

def proposed():
class Cursor:
def __init__(self):
self._iterator = iter(range(10000))

def fetchall(self):
return list(self._iterator)

cursor = Cursor()
cursor.fetchall()

print(timeit.timeit("current()", number=1000, setup="from __main__ import current"))
print(timeit.timeit("proposed()", number=1000, setup="from __main__ import proposed"))
```
resulted in 2.999s and 0.205s respectively, i.e., the proposed solution is about 10x faster.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.