dropbox / dropbox/PyHive

Insert with multiple rows fails

Open
#250 11 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
545
PR merge metrics
No merged PRs in 30d

Description

I use latest PyHive 0.1.8, thrift 0.11.0 and thrift-sasl 0.3.0. I use latest SQLAlchemy 1.2.12.
I use pandas 0.16.0 (the problem is not in Pandas).

I am creating DataFrame with 3 rows. I want to create table if it doesn't exists and put these rows to it.

The DB schema can be defined as following:

```sql
CREATE DATABASE IF NOT EXISTS alx_test;

CREATE TABLE alx_test.alx_score (
ids STRING,
scores_all FLOAT
);
```
The following code in Python 2.7.
Below is simplified version of what I'm trying to achieve:

```python

from sqlalchemy import create_engine
from sqlalchemy.schema import MetaData
from collections import OrderedDict
from pandas.core.frame import DataFrame

dbName = "alx_test"
tableName = 'alx_score'

import logging
logging.basicConfig(format='%(asctime)s [%(process)d]: %(message)s', level=logging.INFO)
logging.getLogger('sqlalchemy').setLevel(logging.DEBUG)

def checkMultiinsert():
hive_engine = create_engine('hive://hive_username@remote_host:10000/default?auth=NOSASL')
meta = MetaData(hive_engine, schema=dbName)
meta.reflect(only=[tableName])

ids_all = ['10', '20', '30']
scores_all = [0.1, 0.5, 0.9]

cols = OrderedDict()
cols.update({'ids': ids_all})
cols.update({'scores_all': scores_all})

df_score = DataFrame.from_dict(cols)

df_score.to_sql(tableName, hive_engine,
schema=dbName,
if_exists='replace',
index=False
)


if __name__ == "__main__":
checkMultiinsert()
```

**Note:** I have to use Hive user in the connection URL because of [bug](https://github.com/dropbox/PyHive/issues/249)

**Note:** I enable logs for SQLAlchemy just to get better understanding of what is going on, you can remove them.

I get following error:

```
Traceback (most recent call last):
File "\poc\multiInsert.py", line 36, in
checkMultiinsert()
File "\poc\multiInsert.py", line 31, in checkMultiinsert
index=False
File "C:\programs\Anaconda\lib\site-packages\pandas\core\generic.py", line 977, in to_sql
dtype=dtype)
File "C:\programs\Anaconda\lib\site-packages\pandas\io\sql.py", line 538, in to_sql
chunksize=chunksize, dtype=dtype)
File "C:\programs\Anaconda\lib\site-packages\pandas\io\sql.py", line 1177, in to_sql
table.insert(chunksize)
File "C:\programs\Anaconda\lib\site-packages\pandas\io\sql.py", line 717, in insert
self._execute_insert(conn, keys, chunk_iter)
File "C:\programs\Anaconda\lib\site-packages\pandas\io\sql.py", line 692, in _execute_insert
conn.execute(self.insert_statement(), data)
File "C:\programs\Anaconda\lib\site-packages\sqlalchemy\engine\base.py", line 948, in execute
return meth(self, multiparams, params)
File "C:\programs\Anaconda\lib\site-packages\sqlalchemy\sql\elements.py", line 269, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "C:\programs\Anaconda\lib\site-packages\sqlalchemy\engine\base.py", line 1060, in _execute_clauseelement
compiled_sql, distilled_params
File "C:\programs\Anaconda\lib\site-packages\sqlalchemy\engine\base.py", line 1200, in _execute_context
context)
File "C:\programs\Anaconda\lib\site-packages\sqlalchemy\engine\base.py", line 1413, in _handle_dbapi_exception
exc_info
File "C:\programs\Anaconda\lib\site-packages\sqlalchemy\util\compat.py", line 265, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "C:\programs\Anaconda\lib\site-packages\sqlalchemy\engine\base.py", line 1170, in _execute_context
context)
File "C:\programs\Anaconda\lib\site-packages\sqlalchemy\engine\default.py", line 506, in do_executemany
cursor.executemany(statement, parameters)
File "C:\programs\Anaconda\lib\site-packages\pyhive\common.py", line 90, in executemany
self._fetch_more()
File "C:\programs\Anaconda\lib\site-packages\pyhive\hive.py", line 380, in _fetch_more
raise ProgrammingError("No result set")
sqlalchemy.exc.ProgrammingError: (pyhive.exc.ProgrammingError) No result set [SQL: u'INSERT INTO TABLE `alx_test`.`alx_score` VALUES (%(ids)s, %(scores_all)s)'] [parameters: ({'ids': '10', 'scores_all': 0.1}, {'ids': '20', 'scores_all': 0.5}, {'ids': '30', 'scores_all': 0.9})] (Background on this error at: http://sqlalche.me/e/f405)
```
As you can see the problem is in hive.py at Cursor._fetch_more() method

This is it's code:

```python

def _fetch_more(self):
"""Send another TFetchResultsReq and update state"""
assert(self._state == self._STATE_RUNNING), "Should be running when in _fetch_more"
assert(self._operationHandle is not None), "Should have an op handle in _fetch_more"
if not self._operationHandle.hasResultSet:
raise ProgrammingError("No result set")
req = ttypes.TFetchResultsReq(
operationHandle=self._operationHandle,
orientation=ttypes.TFetchOrientation.FETCH_NEXT,
maxRows=self.arraysize,
)
response = self._connection.client.FetchResults(req)
_check_status(response)
schema = self.description
assert not response.results.rows, 'expected data in columnar format'
columns = [_unwrap_column(col, col_schema[1]) for col, col_schema in
zip(response.results.columns, schema)]
new_data = list(zip(*columns))
self._data += new_data
# response.hasMoreRows seems to always be False, so we instead check the number of rows
# https://github.com/apache/hive/blob/release-1.2.1/service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java#L678
# if not response.hasMoreRows:
if not new_data:
self._state = self._STATE_FINISHED
```

The reason is in line ` if not self._operationHandle.hasResultSet:` this condition holds, and ProgrammingError is thrown. The state of self._operationHandle is the following:

```TOperationHandle: TOperationHandle(hasResultSet=False, modifiedRowCount=None, operationType=0, operationId=THandleIdentifier(secret='\xac\x9a\x0f\xbf\x83\x87@\x86\xb8\x9e@np\xf8\xf6g', guid='\x81\x17=v.\x83G\xc1\x86U*+\xb4\xca\xa6\xdd'))```

**P.S.** Insertion of 1 row works fine. That is, if I have DataFrame with 1 row, it works. This has to do with SQLAlchemy code (there is check if multiparam has only 1 value, than another execution path is taken, that is one that doesn't involve call to cursor's _fetch_more() method),

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.