dropbox / dropbox/PyHive

Experiencing Exception on Hive/SASL/auth-conf

Open
#397 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
545
PR merge metrics
No merged PRs in 30d

Description

When accessive Hive through SQLAlchemy or the Jupyter Hive Kernel, using Kerberos and the auth-conf setting, we are getting the following exception.

It occurs on a simple select statement listing 300 rows:

```
Traceback (most recent call last):
File "/var/log/hadoop/anaconda3/lib/python3.6/site-packages/sqlalchemy/engine/result.py", line 1211, in fetchall
l = self.process_rows(self._fetchall_impl())
File "/var/log/hadoop/anaconda3/lib/python3.6/site-packages/sqlalchemy/engine/result.py", line 1161, in _fetchall_impl
return self.cursor.fetchall()
File "/var/log/hadoop/anaconda3/lib/python3.6/site-packages/pyhive/common.py", line 136, in fetchall
return list(iter(self.fetchone, None))
File "/var/log/hadoop/anaconda3/lib/python3.6/site-packages/pyhive/common.py", line 105, in fetchone
self._fetch_while(lambda: not self._data and self._state != self._STATE_FINISHED)
File "/var/log/hadoop/anaconda3/lib/python3.6/site-packages/pyhive/common.py", line 45, in _fetch_while
self._fetch_more()
File "/var/log/hadoop/anaconda3/lib/python3.6/site-packages/pyhive/hive.py", line 386, in _fetch_more
response = self._connection.client.FetchResults(req)
File "/var/log/hadoop/anaconda3/lib/python3.6/site-packages/TCLIService/TCLIService.py", line 714, in FetchResults
return self.recv_FetchResults()
File "/var/log/hadoop/anaconda3/lib/python3.6/site-packages/TCLIService/TCLIService.py", line 726, in recv_FetchResults
(fname, mtype, rseqid) = iprot.readMessageBegin()
File "/var/log/hadoop/anaconda3/lib/python3.6/site-packages/thrift/protocol/TBinaryProtocol.py", line 134, in readMessageBegin
sz = self.readI32()
File "/var/log/hadoop/anaconda3/lib/python3.6/site-packages/thrift/protocol/TBinaryProtocol.py", line 217, in readI32
buff = self.trans.readAll(4)
File "/var/log/hadoop/anaconda3/lib/python3.6/site-packages/thrift/transport/TTransport.py", line 60, in readAll
chunk = self.read(sz - have)
File "/var/log/hadoop/anaconda3/lib/python3.6/site-packages/thrift_sasl/__init__.py", line 173, in read
self._read_frame()
File "/var/log/hadoop/anaconda3/lib/python3.6/site-packages/thrift_sasl/__init__.py", line 187, in _read_frame
message=self.sasl.getError())
thrift.transport.TTransport.TTransportException: b'Error in sasl_decode (-1) SASL(-1): generic failure: Unable to find a callback: 32775'
```

Turns out this is generated by the Cyrus-SASL native library.
I am able to fix the issue by specifying a larger value for 'maxbufsize' in the sasl_factory in hive.py. The default is 65535 bytes or so.

```
def sasl_factory():
sasl_client = sasl.Client()
sasl_client.setAttr('host', host)
if sasl_auth == 'GSSAPI':
sasl_client.setAttr('service', kerberos_service_name)
#+B. Gagnon 2021-07-12: patch to support auth-conf
sasl_client.setAttr('maxbufsize', 2*1024*1024)
#-B. Gagnon 2021-07-12: patch to support auth-conf
elif sasl_auth == 'PLAIN':
sasl_client.setAttr('username', username)
sasl_client.setAttr('password', password)
else:
raise AssertionError
sasl_client.init()
return sasl_client
self._transport = thrift_sasl.TSaslClientTransport(sasl_factory, sasl_auth, socket)
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.