Possible DB-API fetchall() performance improvements
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 545
- PR merge metrics
- No merged PRs in 30d
Description
Currently the cursor implementation for [fetchall](https://github.com/dropbox/PyHive/blob/65076bbc8697a423b438dc03e928a6dff86fd2cb/pyhive/common.py#L129) indirectly iterates over the result set via `fetchone` rather than directly iterating over the result set similar to prestodb's python [DB-API](https://github.com/prestodb/presto-python-client/blob/master/prestodb/dbapi.py#L306).
I'm not certain whether it's viable to use this approach, however I mocked up a basic example which shows that the proposed solution is about 10x faster.
```python
import timeit
def current():
class Cursor:
def __init__(self):
self._iterator = iter(range(10000))
def fetchone(self):
try:
return next(self._iterator)
except StopIteration:
return None
def fetchall(self):
return list(iter(self.fetchone, None))
cursor = Cursor()
cursor.fetchall()
def proposed():
class Cursor:
def __init__(self):
self._iterator = iter(range(10000))
def fetchall(self):
return list(self._iterator)
cursor = Cursor()
cursor.fetchall()
print(timeit.timeit("current()", number=1000, setup="from __main__ import current"))
print(timeit.timeit("proposed()", number=1000, setup="from __main__ import proposed"))
```
resulted in 2.999s and 0.205s respectively, i.e., the proposed solution is about 10x faster.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.