Support for kerberos HDFS clients
- Dominant language
- Python
- Stars
- 1k
- Forks
- 131
- PR merge metrics
- No merged PRs in 30d
Description
The cluster I have access to uses kerberos authentication exclusively, and so I've needed a kerberos client in order to use odo's HDFS functionality.
The [hdfs](https://pypi.python.org/pypi/hdfs/) package has both kerberos and avro extensions.
Something like
``` python
import hdfs
from hdfs.ext.kerberos import KerberosClient
class _KHDFS(object):
""" Parent class for data on Hadoop File System
Examples
--------
>>> KHDFS(CSV)('/path/to/file.csv', host='54.91.255.255',
... port=14000, user='hdfs') # doctest: +SKIP
Alternatively use resource strings
>>> resource('khdfs://hdfs@54.91.255.255:/path/to/file.csv') # doctest: +SKIP
"""
def __init__(self, *args, **kwargs):
hdfs = kwargs.get('hdfs', None)
host = kwargs.get('host', None)
port = kwargs.get('port', None)
if not hdfs and (host and port):
url = "http://{host}:{port}/".format(host=host,port=port)
root = "/"
hdfs = KerberosClient(url, root=root)
if not hdfs:
raise ValueError("No WebHDFS url found:.\n"
"Must supply keywords\n"
" host=, port=")
self.hdfs = hdfs
kwargs.update(hdfs=hdfs)
self.subtype.__init__(self, *args, **kwargs)
```
I've got a functional version of this in a branch, but I'm doing some hacky things that I wouldn't put in a PR.
1. `KerberosClient`'s API is materially different than the `PyWebHdfsClient` object in odo, so I ended up writing parallel implementations using `KerberosClient` of most of the methods in `odo/backends/hdfs.py`
2. In order to allow the two implementations to coexist, I used a different resource pattern `khdfs://user@ip:port:/path/to/file.csv`. This is not nice at all, and it would be better to be able to configure which implementation is used rather than expect the user to change resource URLs.
I'm wondering if this is valuable to anyone else, and if anyone has recommendations about how to solve these two issues.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.