dropbox / dropbox/PyHive

Pandas DataFrame to Hadoop using Hive and Impala uses incorrect data type

Open
#291 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
545
PR merge metrics
No merged PRs in 30d

Description

# In short, issue is that SQLAlchemy seems to be trying to create following, and error is that type TEXT should be type STRING or VARCHAR

CREATE TABLE test_pandas_to_hive (
index BIGINT,
name TEXT
)

# Code starts here

import pandas as pd
from sqlalchemy import *
from sqlalchemy.engine import create_engine
from sqlalchemy.schema import *

# Connection to Impala Hadoop
engine = create_engine(
'impala://',
connect_args={'port': ,
'auth_mechanism': 'GSSAPI',
'database': 'my_db_scratch_space'},
)

# Simple dataset
df = pd.DataFrame({'name' : ['User 1', 'User 2', 'User 3']})

# Now write to Hadoop
df.to_sql('test_pandas_to_hive', con=engine)

# Get the following error, seems to be trying to create a table in Hadoop, but the data type is wrong
# Is trying to create a column of data type 'text', which doesn't exist
# Should be of data type 'string' or 'varchar' instead

---------------------------------------------------------------------------
HiveServer2Error Traceback (most recent call last)
~/.conda/envs/Python3venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py in _execute_context(self, dialect, constructor, statement, parameters, *args)
1243 self.dialect.do_execute(
-> 1244 cursor, statement, parameters, context
1245 )

...

~/.conda/envs/Python3venv/lib/python3.6/site-packages/impala/hiveserver2.py in err_if_rpc_not_ok(resp)
702 resp.status.statusCode != TStatusCode.SUCCESS_WITH_INFO_STATUS and
703 resp.status.statusCode != TStatusCode.STILL_EXECUTING_STATUS):
--> 704 raise HiveServer2Error(resp.status.errorMessage)
705
706

DBAPIError: (impala.error.HiveServer2Error) AnalysisException: Syntax error in line 4:
name TEXT
^
Encountered: IDENTIFIER
Expected: ARRAY, BIGINT, BINARY, BOOLEAN, CHAR, DATE, DATETIME, DECIMAL, REAL, FLOAT, INTEGER, MAP, SMALLINT, STRING, STRUCT, TIMESTAMP, TINYINT, VARCHAR

CAUSED BY: Exception: Syntax error

[SQL:
CREATE TABLE test_pandas_to_hive (
index BIGINT,
name TEXT
)

]
(Background on this error at: http://sqlalche.me/e/dbapi)

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.